1: Rewrites and Refactors

Darby talks about his new idea for a Lead Honestly rewrite and Ethan has a unicorn of a refactor.

Ethan: [00:00:00] Good afternoon. Okay.
Darby: [00:00:02] Happy Friday. Don't normally do this on Friday, but it's kinda nice.
Ethan: [00:00:06] It is a little nice. 
Darby: [00:00:07] Are you done with the day job for the week?
Ethan: [00:00:10] no, I'll have probably another hour after we're done here. But it's Friday. So that's a late day anyways.
Darby: [00:00:16] Yeah. Things get real quiet for me around Noon.
Ethan: [00:00:19] Is that because most of your team is overseas?
Darby: [00:00:22] Yeah. Yeah.  A of, A lot of people kinda wrap up fairly early.  There's usually some stuff that trickles in, but it's much quieter than the rest of the days,
Ethan: [00:00:30] Yeah.
Darby: [00:00:30] I don't mind it. Yeah. So I'll go first today.
Ethan: [00:00:34] Okay.
Darby: [00:00:36] Last week we were talking about My idea this Google docs idea that I was working
on. So I had, I don't remember how this started, but my brain just started turning on this idea. And I was like, if we, so we have this application that's running, it's been running for a few years and it's got a lot of legacy assumptions built into it. What if we started over.
Ethan: [00:00:59] Ooh.
Darby: [00:01:01] And so
Ethan: [00:01:02] Tell me more.
Darby: [00:01:02] started thinking about it and I was like, this is what base camp does.
They did base camp to base camp three. They're all like kind of rewrites. And they like leave. I think I don't really know how their process works, but I know that you can go to old versions and still access them. And if you don't want to upgrade, maybe you get like security updates or something and that's about it.
So I was thinking our customer base is not that large. And the way that I see this moving in the future is much more like integrated with the tools that the customers are using instead of trying to get them to shift their behavior to a different tool. So in our case, make the Google docs better for them instead of trying to drive them somewhere else.
And so if I think about that, we start over. Instead of having like a regular sign up or sign in, you just use Google auth because you're going to be using Google docs, or if you're using say Microsoft products or whatever, and then it's, it simplifies a ton of other things. Like once you get past that, because then you don't have if we're doing it this way, here's the code for doing it that way.
Here's the code. It's just there's just kind of one way to do it. Cause there's, we're not building for a whole bunch of different use cases. And so I I rails nude on I dunno, what was it Saturday last week. And so I've got like the full prototype working from like sign-in hook up your Google calendar, hookup, your Google doc, put some data in there, show a dashboard, no styles at all.
Of course, but just like the whole thing working. I got that done yesterday. So what was that like six days?
Yeah, a ton of assumptions, ton of, no tests, none of that. But wanted to prove out that the whole thing could work and like I'm going to show to share this afternoon. But I think there's actually an interesting approach to this, that that I want to explore a bit.
Ethan: [00:02:50] That's really, that is really interesting. So one of the, one of the things we talked about last time was. The idea of this lighter version of lead, honestly, that dumps into a Google doc is as a form of expansion revenue. Like this would be potentially easier to get people to buy that version and then they could grow into the more full-fledged would you still do that with two separate apps, or would you look at it as like actually two separate products.
Darby: [00:03:21] Yeah, I think that was my thinking last time. And now I feel like I wouldn't even want to like, try to push them to something else because I think it fundamentally, the tool that you're working in for most of the work that you're trying to do, needs to be like the most powerful thing for that action or for that task.
And so I think like a one-on-one doc focused around a Google doc or a one-on-one experience focused around a Google doc, it gives you a ton of flexibility, right? Like you can both type at the same time and you can see the cursors and you can highlight, and you can make notes and comments and format text and, all kinds of things. And to be able to replicate that experience in a different tool, it would just take tons of engineering effort to get there.
Ethan: [00:04:02] Look how much engineering effort we put into the Wiziwig editor? That's just a small component of what you get out of Google docs. Yeah.
Darby: [00:04:11] Yeah so I started thinking, yeah. Started thinking along these lines, it's like, all right let's leverage the tool. That people want to use any way make that experience work really well. And then add on top of that, like all of the capabilities that we want to have. Really what the value that, that I think we bring to the, like this manager, employee relationship is all of the coaching.
The questions that we can ask, the suggestions that we have like this is I dunno at the lowest level, there are pieces of content and that we show to a user at the right time. Like we should make that really good and not spend a bunch of time building like a WYSIWYG editor. So that's like where my head's at with it.
And then I started going crazy. Cause I was like if we did this like instead of again, spending a bunch of time building a wizzy wig editor, we could. Make an app that could be, like a true, like single page app for the features that we're gonna take a user to in our application and then just make that an electronic app, or maybe make that a mobile app.
Like you could spend time on those things instead of reinventing the wheel. Which I feel like we, we spend a ton of time on, 
Ethan: [00:05:13] all right. So you're gonna show that to Shay today.
Darby: [00:05:16] Yeah. I'm chatting with him right after this. 
Ethan: [00:05:18] Okay. I'm interested to see what his take on it is or hear what his take
Darby: [00:05:22] yeah. Yeah. It's as I started rewriting things, I was like, yeah, there's a lot of stuff that we'll have to like, port over like all of our questions, our whole question model and the whole playbook thing. Like we would still want that in some way, but maybe not the same way that we have it today.
Maybe we could simplify it. And then our pulse surveys, like that's a whole other feature that is I don't think that would really change that much. So do we duplicate that? Probably not that hard to do it but it's this weird kind of middle ground. It's not super, obviously we can just throw away all the old stuff and start over, but I think it would like we'd we get to. Where we want to get to with it much quicker, because we wouldn't be dealing with all the legacy considerations. 
Ethan: [00:06:01] Yeah. That's really interesting. It's definitely at least worth exploring, would you leave the you'd leave the existing lead honestly, up and running.
Darby: [00:06:11] yeah, probably at least for some period of time, maybe give people a migration path if they want. But for some people that won't work because they're not on like Google,
Ethan: [00:06:19] So when you leave signups open on it.
Darby: [00:06:22] Probably not. Like until we got the new one done 
Ethan: [00:06:26] I didn't know how far you wanted to go, like the parallel track or this really is like the base camp approach of you're on base camp. Two, we'll leave that forever, but all of our energy and attention is on subsequent versions.
Darby: [00:06:41] I think, yeah, I think probably that's what I would say would be the right way to go. And mainly because I don't want to confuse people during sign-up like, you just want it to be like, here's what you click on, just do this. And then you get the thing instead of having to understand what is version one and what is version two and why would I want one or
the other,
Yeah. Yeah, it was fun to, to start hacking on stuff again. There's the complexity is of like Google API is there's some, yeah. There's a lot going on there. Yeah. Yeah. So that's that's what I've been up to. 
yeah, that's about it. 
Ethan: [00:07:19] cool. I had a pretty good week. This week after we talked last week, I pretty quickly got the deployment stuff figured out so I can deploy to Heroku. Which, so I haven't used Heroku since we were at belly together. been five years, it has gotten so much better. I I think I was a little jaded on Heroku based off of.
We were early alpha users of private spaces.
And so there was a lot of rough edges there. I don't know if you remember, we had the, we ran out of space in our private space sub-net so we couldn't spin up any more. Dynos do 
Darby: [00:08:00] I, yeah, I think I do.
Ethan: [00:08:01] And so they, yeah they they were going to resize the, what I'm assuming is the underlying VPC for us and subnets.
It was supposed to be like a seamless thing and ended up being almost eight hours of downtime,
  
Which is not fun at all. I haven't really used use them sense but getting I already had a Docker file set up for my existing Kubernetes deploy. And so just telling Heroku, like where to find the Docker file. They build the image or they build the container and then launch it.
And then you can even do post and pre deploy steps. So like they'll run migrations. And if the migrations don't pass, they stop the deploy Oh, like just cool stuff like that is built in which wasn't there last time. I don't know. I'm really impressed for $7 a month for my dyno. It is 
Darby: [00:08:52] nice. you're not using private spaces.
Okay. Okay. And I'm curious if that's gotten better. That's how that's always the thing that like bugs me about Heroku is the, like the, your database has to be world accessible. I wish there was a quick fix for that, but 
other than that, I think it's got 
Ethan: [00:09:11] Yeah, it's so nice. I have not been giving them enough credit for sure. I am using their database though, so I'll probably migrate off of it. When I actually launched the product, but for now I don't care.
Darby: [00:09:24] I I know the answer to this, but why would you want to migrate off of it?
Ethan: [00:09:27] Cost would be the big thing. 
Yeah. Yeah. Yeah. RDS hosting, even with setting up like a multi-age Z cluster is very cheap. That being said maybe I won't though, maybe I'll get to launch and decide that it's good enough for now.
Darby: [00:09:47] I think the the, we have the same setup, so Heroku with RDS. And I think the I've heard stories that once you go to all in, on, on the. Heroku database. It's hard to move if you want to. And so I think hearing those stories, I felt okay, 
enough just to do this in AWS 
Ethan: [00:10:06] Yeah. Yeah. Yeah, we'll see you. So I got that done. I think funny enough, I'm still running the Kubernetes cluster. I should shut that down. But then, so we talked a little bit about this last week. I was running into issues with essentially like default scopes. Like how can I make sure that the data that I'm accessing is. The scope to the right user and being like really paranoid about this because it's financial data. Like I definitely don't wanna leak someone's finances to a random user. It turns out like maybe two weeks before I was thinking about this problem the elixir and ecto team echo is the database library wrote a whole guide on how to do this. And it is it's so well done. It's fantastic. So I implemented that. And as part of that I also had to change my data model, which made, actually made my data model easier to use. 

the way that it yup. So the way that glean is set up is there's this top level model called household and then users belong to a household and then accounts hang off of the household and then accounts have balances.
And they're rolled up into like institutions. So like I have three accounts at chase. That belong to me. So like the issue I was running into is if I wanted to display all of the transactions across a household, you have to start at the top and then join your way down, like through the object graph to get all of the transactions.
So like basically, like you always have to start at the top. You always have to say, I have a household now join in everything downward. Just everything becomes like a little bit more complicated than what you would want it to be. And so the way that this is set up now is every object in the system has a household ID.
So every object, every record knows that it belongs to a household.
 So I can say every query that runs through the system has to have in household like you. So it's not possible to query for data without providing the household ID.
So that solves the security problem. Like it's, you can't forget to add it. It just won't work if you don't scope it, which is fantastic.
And because everything has a household ID, I can just go and say, give me all the transactions for this household and totally ignore the rest of the object graph.
Yeah.  
Darby: [00:12:31] Nice.  I suppose the thing you can't do then is like easily move. Individuals between households or something like that. If you weren't structuring the data, that way you, things would be a bit more independent, but 
Ethan: [00:12:47] Yeah. So you mean 
if you have a bunch of account data that you want to move to a different, 
Darby: [00:12:52] Yeah. 
I don't know. 
Ethan: [00:12:54] but that's a good, that's a good example. Yeah, there would be. It's possible. You just essentially have to open a transaction and update
all of it. All of it at the same 
Darby: [00:13:05] Yeah. Yeah. the primary use case either.
Ethan: [00:13:12] no. 

Yup. Yup. I don't know. I think it's, I don't know if I've had a big refactor like this, where the end result is a more secure system, but it's also easier to use, like that almost never happens. It's I got the systems more secure, but because of that, there's like more Gates in place or, more process or whatever it may be.
It's never that, Oh, I could query data significantly easier now in a more secure way, like that's. 
Darby: [00:13:44] I think that's like the pattern of thinking with relational databases. I know I do this myself all the time is it's okay, like you have this very like cleanly structured relational data system. And then you're like, cool, that's great. Let's ship it. And then you go to query and you're like, Oh, this is a nightmare.
Exactly. And I think that's a really good example of why wouldn't you just do this? Like, why wouldn't you just put a user ID or an account ID on all of your objects then when you need to search it, you're not like relying on the correctness of the data model that 
 Ethan: [00:14:20] 
The other fun part too, is it uses Postgres as composite foreign keys. The account balance belongs to an organization. It also belongs to an account and that composite foreign key ensures that they both agree on the household ID. So the end result is since everything has a household ID and they all have to agree that they belong to the same household ID, you get that referential integrity across the whole object graph. So it's not like you can't like accidentally insert to the wrong household. It just, 
Darby: [00:14:56] Yeah. 
That's nice. Yeah. So that's enforced at the database level, so you don't even have, you'll build your validations and everything too, but 
Rigid 
Ethan: [00:15:04] Yup. Yup. That was fun. And then I ended trying to figure out what Java script as most things. And yeah, I wanted to get some charts built up. So you can try the first chart is going to be your total net worth over time. So now that I have all the accounts with their balances and those balances come in daily I can start charting that.
So it's trying to figure out what charting library to use, how to hook it up into the asset pipeline and all the stuff that I've been ignoring so far.
Darby: [00:15:37] okay. Yeah. That's 
Ethan: [00:15:39] Yeah. Yeah. I laughed when I started Googling for a charting library. So it was like JavaScript, charting library, and then Google automatically filled in 2021. So it's like that's how, like even Google knows how fast this 
Darby: [00:15:59] right? Yup. Yup. 
 
Yeah. It sounds like you had a really good start and then  
Ethan: [00:16:06] So the chart displays my math is wrong somewhere. It, like it was showing a net worth of 800 million. Which is, I don't think. Correct. So I got to figure out where I messed up my math on that.
Oh man. Yeah, I think it, I'll figure it out for sure. But I'm guessing I store all 
values as cents. I'm somewhere. I'm probably not converting that cents back into dollars. And then assuming that it already is dollars. 
Darby: [00:16:34] Yeah. 
Yeah.  that can do it. 
Is there, 
I know in Ruby there's some pretty like good libraries for this sort of thing. Is, 
Ethan: [00:16:44] yep. 
No. There's a good one. It's called money. It even, it does the hard work to have, or not hard work. The boil boiler plate work of you give it a dollar amount. It knows how to change that dollar amount into cents. And then it stores that in the database. It's I can tell Ecto that this is a money type.
And it knows to use that library to dump in load values and even handles currency conversion. So it should be pretty easy to handle things besides USD. So it'll know if it's Euro where like it use commas instead of periods 
Darby: [00:17:21] Okay. 
Ethan: [00:17:21] stuff. 
Darby: [00:17:21] rates,
Ethan: [00:17:24] Not currency 
rates. 
No, just
currencies. 
Darby: [00:17:26] impressive.  
Ethan: [00:17:28] No, 
Darby: [00:17:28] don't want that.
Ethan: [00:17:30] And next steps for me, I'm going to try to wrap up the charting library work so that, that is accurate. Maybe design it a little bit, and then I'm going to pull in transaction data. So that'll be a relatively big chunk of work. So it's when you bring in an account into the system, try to pull as much individual transactions as I can so that I can build up spending patterns and stuff like that with historical data, but then also getting the web hook library set up for as they come in.