• Wow.

    I’ve been processing a couple of billion rows of data on my machine, the fans didn’t even come on. WTF are they teaching “experts” these days, or has Elmo only hired people who claim that they can “wrangle data” and say “yes” ?

    • bleistift2 ( bleistift2@sopuli.xyz ) 
      link
      fedilink
      English
      arrow-up
      135
      ·
      2 years ago

      Even if querying data was processing-heavy and even if somehow the ‘hard drive’ got warm during this, then there still would need to be a hardware defect in order for the drive to overheat.

    • has Elmo only hired people who claim that they can “wrangle data” and say “yes” ?

      There’s two issues going on:

      1. Elmo’s sociopathic approach to laying people off is public knowledge, and top experts have the luxury of not even applying for his jobs.
      2. Elmo’s ability to judge engineering talent has likely been wildly exaggerated thanks to how he has successfully bought organizations full of talented people, in the past.
    • 60k rows is generally very usable with even wide tables in row formats.

      I’ve had pandas work with 1M plus rows with 100 columns in memory just fine.

      After 1M rows move on to something better like Dask, polars, spark, or literally any DB.

      The first thing I’d do with whatever data they’re running into issues with is rewrite it as partitioned and sorted parquet.