Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Report Developers, on the other hand, are folks who have made a career around designing reports in a specific tool (e.g. Microstrategy, et al). They are specialists.

Is this the common perception, because it really doesn't line up with my experience?



At least in my Org Reports are pretty much an after thought left to the data engineers (like me) to "take this metric I've developed" and display it on the morning report.

Writing/updating a report is easiest part of my job it's the data that goes into building it that is hard translating the "simple metric I've developed" and getting it to run in a robust automated and sane fashion is the difficult part.

The complexities in my org are two fold.

Firstly the infrastructure people don't get data - at all. They speak PLC's and HMI's to them it's all OPC and magic A2A messaging takes care of everything. All data is time series to them and it all goes into an historian (which is basically a giant ring buffer i.e it gets flushed periodically) anything beyond that is past their level of expertise.

The data needs to be batched together the time series information has to be processed into "event frames" - this data was all part of this sequence of conveyor belt movements for example. Then you need to link it to related events etc and archive it in some kind of sane fashion so that in six months time if there is a product defect or something like that you can trace the entire series of event frames for that particular production batch.

Secondly the people the article calls "data scientists" (in my org these are Engineers - real ones of the Chem and Mech variety) don't know anything about databases or handling data they prototype their metrics in Matlab, Fortran, Excel and the like.

You really need someone to translate their code into something sane that can be automated. Engineers are not taught to code at all. I know I studied engineering at university Fortran is the lingua franca. Code is just a way of representing mathematics. Asking these people to do all the data processing pipeline is just not going to happen. It's not their job. They write the simulations and models they have the domain knowledge thats whats important for them to be worrying about.


Ok, I think I'm getting the specifics of this situation. So, we are talking about internal reports, not something that could actually get in the external customer's hands.


Yes this is internal stuff. I work at a large industrial manufacturing plant.

Reports that go externally are done by certified people. (Laboratory technicians for product specifications and finance analysts for stock market stuff).


I’ve done external reports for clinical trials and agriculture, and I guess they weren’t as up on getting certifications. Thanks for the very detailed replies.


Seems like the author is talking about the pre-big data version of business intelligence with star schemas and attempts at drag and drop tools, which has been supplanted somewhere around 2010-2015 by open source big data tools. I wasn't at a big enough company to have a proper BI department pre the data science renaming, so I can't really opine on whether it's true.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: