![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
chipotleI started this entry sitting in the cafeteria of my workplace trying to find a more comfortable place to work on stuff without being handed more things--not that I have a particularly huge workload in terms of items, just that what I have is getting difficult to handle. Knowing that my coworker (the one I was hired on to help) is actually doing a lot more than I am is not a source of real comfort.
Back in a previous life, I liked using Excel for data analysis. No, really. It's a great tool for small to moderate data sets. If you know what you're doing you can use it with large data sets, too, provided you have some level of control over the back end--the actual data store, or "data warehouse." But in this case, I don't, and in this case the volume is probably too large. One table in the data warehouse has over fifty million rows. Fifty million.
The data warehouse of the company isn't organized the way I would have organized it--although granted, my view on "the right way" is influenced by the reporting needs I'm running into, which might not have been predictable originally. (Although some of it probably could have, and indeed should have.) I'm told that canonical data warehouse theory all but stands canonical database theory on its head: it's good to denormalize data rather than normalize it, and--unlike the reporting server in that past life--you don't want to simply mirror the actual production database. That part I can understand; the denormalization I can't, and the near lack of indexing drives me nuts. And since I'm a read-only user of the reporting server, there's nothing I can do about that, save put in requests with the folks in control--who are already far more overloaded than I am. ("Indexes" on databases greatly speed up access to them, but you probably won't know fields in a database to index unless you're designing the reports--which means that to run an efficient ship, you either need someone in the database team dedicated to reporting tools, or you need to give someone in the reporting team levels of access which will scare the database administrator.)
As much as the thought will horrify some folks, I'd love it if the warehouse ran Microsoft SQL Server rather than MySQL. No, it's not quite as fast, but it's a lot more powerful, and darned if it doesn't work better with Microsoft products. See, my day is increasingly consisting of starting queries, letting them run and lock up whatever application I'm using until either they're finished or I can't or won't wait any longer for them. And, increasingly, it's "can't," not "won't": more requests for reports come in and they crash against the unyielding rocks of the unresponsive database server.
And you know, not being able to do your job kind of sucks. It can make you look incompetent and it certainly makes you feel incompetent.
Beyond that, the timing is getting to me. Not the commute, exactly--the commute's annoying but honestly less so than the commute to NetPoodles was. No, it's the hours of 10:00 to "sometime after 7:00." When being home by 8:00 is early, it's difficult to get into bed by midnight, let alone before it. Making it in bed by 1:00 doesn't seem late--but of course it means that I'm highly unlikely to be out of bed before 8:00. Unless I sequester myself in my room, I'm not likely to be doing much creative in the evening, yet it's very difficult to get up early enough to take advantage of morning energy.
Of course, I could try to shift my work hours--but the chances are leaving at the "early" hour of 6:00 every day would be looked at askance regardless of my arrival time. (At NetPoodles, I left at 5:30.) And of course, normal working hours would make the commute worse.
Well. It's still a great place to work, and I'd still be perfectly happy if my contract--which ends again in less than two weeks--gets renewed a second time. But I wish it was a lot closer to where I live.
I'll keep trying to find the time to attend to "personal" projects, including that elusive goal of finding freelance writing work. The idea of being my own boss is increasingly attractive, and it's not quite as scary as it used to be--perhaps because I'm doing 1099 work now, perhaps because I'm no longer nervous--well, as nervous--about having my income radically cut. (A cut to zero would be bad, obviously.)
Well. It's well past midnight now, so I'll take another small step in that apparently fruitless struggle to get up early enough to provide personal time: I'll go to bed.
Back in a previous life, I liked using Excel for data analysis. No, really. It's a great tool for small to moderate data sets. If you know what you're doing you can use it with large data sets, too, provided you have some level of control over the back end--the actual data store, or "data warehouse." But in this case, I don't, and in this case the volume is probably too large. One table in the data warehouse has over fifty million rows. Fifty million.
The data warehouse of the company isn't organized the way I would have organized it--although granted, my view on "the right way" is influenced by the reporting needs I'm running into, which might not have been predictable originally. (Although some of it probably could have, and indeed should have.) I'm told that canonical data warehouse theory all but stands canonical database theory on its head: it's good to denormalize data rather than normalize it, and--unlike the reporting server in that past life--you don't want to simply mirror the actual production database. That part I can understand; the denormalization I can't, and the near lack of indexing drives me nuts. And since I'm a read-only user of the reporting server, there's nothing I can do about that, save put in requests with the folks in control--who are already far more overloaded than I am. ("Indexes" on databases greatly speed up access to them, but you probably won't know fields in a database to index unless you're designing the reports--which means that to run an efficient ship, you either need someone in the database team dedicated to reporting tools, or you need to give someone in the reporting team levels of access which will scare the database administrator.)
As much as the thought will horrify some folks, I'd love it if the warehouse ran Microsoft SQL Server rather than MySQL. No, it's not quite as fast, but it's a lot more powerful, and darned if it doesn't work better with Microsoft products. See, my day is increasingly consisting of starting queries, letting them run and lock up whatever application I'm using until either they're finished or I can't or won't wait any longer for them. And, increasingly, it's "can't," not "won't": more requests for reports come in and they crash against the unyielding rocks of the unresponsive database server.
And you know, not being able to do your job kind of sucks. It can make you look incompetent and it certainly makes you feel incompetent.
Beyond that, the timing is getting to me. Not the commute, exactly--the commute's annoying but honestly less so than the commute to NetPoodles was. No, it's the hours of 10:00 to "sometime after 7:00." When being home by 8:00 is early, it's difficult to get into bed by midnight, let alone before it. Making it in bed by 1:00 doesn't seem late--but of course it means that I'm highly unlikely to be out of bed before 8:00. Unless I sequester myself in my room, I'm not likely to be doing much creative in the evening, yet it's very difficult to get up early enough to take advantage of morning energy.
Of course, I could try to shift my work hours--but the chances are leaving at the "early" hour of 6:00 every day would be looked at askance regardless of my arrival time. (At NetPoodles, I left at 5:30.) And of course, normal working hours would make the commute worse.
Well. It's still a great place to work, and I'd still be perfectly happy if my contract--which ends again in less than two weeks--gets renewed a second time. But I wish it was a lot closer to where I live.
I'll keep trying to find the time to attend to "personal" projects, including that elusive goal of finding freelance writing work. The idea of being my own boss is increasingly attractive, and it's not quite as scary as it used to be--perhaps because I'm doing 1099 work now, perhaps because I'm no longer nervous--well, as nervous--about having my income radically cut. (A cut to zero would be bad, obviously.)
Well. It's well past midnight now, so I'll take another small step in that apparently fruitless struggle to get up early enough to provide personal time: I'll go to bed.
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
no subject
Date: 2003-04-30 08:52 (UTC)No indexing on tables?... That's not good for queries. And what's the normalization you speak of?
no subject
Date: 2003-04-30 11:07 (UTC)no subject
Date: 2003-05-01 08:51 (UTC)There are indexes on tables, but they're not necessarily the right indexes. The aforementioned 50M+ record table has a timestamp which it's not indexed on, for instance. Evidently there's another field in it with a non-obvious name that has a Unix timestamp in it which *is* indexed.
The "normalization" is the canonical rule of not duplicating data within a database. Instead of having one table that lists your CD collection with "CD title," "artist" and "song" fields, you have three tables that assign them unique keys, so every song only appears once in the song table and every artist only appears once in the artist table, even though you might have five Alan Parsons CDs or three different artists doing versions of "Let It Be." Evidently there's a theory in data warehousing that it's okay to have "denormalized" tables--taking those three and just dropping them back into the one huge CD table full of duplicates--as part of abstracting the warehouse from the production database. I can certainly see the value in abstraction, so changes to the production database only require changes in the script that updates the warehouse. (At Intermedia their "reporting server" was just a mirror, although in practice I don't think the database made any changes other than adding fields and tables in the five years or so I was working with it.) But I'm not sold on the value of denormalization. I suspect it also creates part of the problem with the indexes--when you're setting up a table that combines five other tables into one big denormalized chunk, it may not be at all obvious what is and isn't a good indexing field.
no subject
Date: 2003-05-01 09:12 (UTC)Also note that the way I read the contract, since it doesn't specify the hours I work on a given day, if I come in at 1:00 p.m. I'm not going to deduct anything from my invoice. The silver lining is that, within reason, I can set my own hours. It's clear I don't have the startup mindset, though; I'm not going to be "married to my job" unless my job is writing for myself, or something somewhat off-the-wall like park service ranger. (I've considered a month-long volunteer position as back country ranger in Yellowstone, but I'm not sure I'm really up to that yet!)
The two things that hold me back from simply setting my own hours earlier are the commute I mentioned, which is about what the commute to NetPoodles was only by virtue of the lighter traffic after 9:00 a.m. and 7:00 p.m. (those hours being when the carpool lane opens up to single-passenger cars, too), and the strong suspicion that if I managed to make it in at 8:30 a.m., I'd still end up staying until 7:00 p.m. simply because the rest of the office would just breeze past the logical 5:30 p.m. stopping point and drag me along with it. I've realized that this really comes from a "we're building something really cool and investing all our energy into it" mentality that I suppose permeates any business venture that's still in its formative stages (at least ventures that it's possible to get passionate about). I just strongly suspect that the only "ventures" that would get me to commit insane hours to them would be ventures either entirely for me (like writing) or living where you work (like a ranger)--essentially, something that I loved sufficiently that the line dividing work from non-work blurs.
no subject
Date: 2003-05-01 09:45 (UTC)As for the database, well, I don't follow the use of normalization then; if you have a denormalized table with all the information in it, you can search on it in any fashion you desire, and you can index it by cd title, artist, and song. Unique keys are to enforce database consistency, so only one item of that kind can exist in that table and should be updated rather than having another row inserted.
But of course, having to search through a bazillion rows and getting slow results indicates that it's not scaling well, and a different approach should be used.
no subject
Date: 2003-05-02 07:47 (UTC)What would happen if you came in earlier and simply said, when 5:30 hit, "see you all tomorrow" whether they were still engaged or not? Would they resent you not staying late?
Have a great weekend!