Parallel list iteration?

bsg75 · 3 days

My Python script is actually using dict(zip(k, v)). This is an educational experiment in Lua, and a way to compare performance (LuaJIT).

I have data in csv files without a header line. In the Python script, the list of fields is kept in a list, and each line of the csv file is split into a list, matched with dict(zip(k, v)), and sent to Redis.

It is a very simplistic case - there is no quoting of the data, and there is an expectation that each csv line will have the correct number of fields.

zacharya15 · 6 days

Having moved from a MSSQL DW/BI role to one using PostgreSQL, I would offer the opinion that it is definitely possible with Postgres, but there is one limiting factor - the engine cannot parallelize query execution.

This causes some of my OLAP workloads to become CPU bound, without the effort of manually partitioning and aggregating the work. PostgreSQL has most of the SQL constructs needed, and work on parallel execution is in planning.

SymetriX · 8 days

App devs love ERDs.

SymetriX · 8 days

I had not noticed your username before thinking of some evil Matrix system of control looking at that diagram.

bsg75 · 8 days

Works perfectly. I have gotten so used to "batteries included" I forgot basic structures ;)

bsg75 · 8 days

This particular case has very simple data, where most CSV parsing would need a more generic search pattern. But now I know about Pen/Microlight. TYVM.

bsg75 · 8 days

Looking at the LuaCsv example, I see it deals with quoting - necessary in a generic csv parser, but in my case there is no quoting.

I am looking for a simple string.split implementation, since in a performance comparison that is what I am testing in Python.

Currently digesting http://lua-users.org/wiki/SplitJoin

Edit: Perhaps as simple as "for b in string.gmatch(a,"%w+") do" ?

bsg75 · 8 days

Easy / concise is what I am after. My Python test simply splits the sting of columns on ',' into a keys list, splits each csv row into a values dict, and zips them into a dict, which is then sent to Redis.

An equally simple Lua script make for a good educational exercise, and good performance test, considering both developer and execution velocity.

bsg75 · 12 days

Why JavaScript?

bsg75 · 12 days

I'd say learn Lua because it's an interesting language

Agreed, and I can get something useful implemented in a shorter timeframe. The time I can set aside for education not in support of an immediate deliverable is limited, and coming from Python, I will likely grasp Lua quicker.

First experiment: Convert a Python script to Lua, that pivots CSV file data into Redis protocol (and figure out the most efficient way to string.split()). Mostly a runtime performance comparison and academic exercise.

C is still on the ToDo list, so much of the OSS software I work with is in C, and if I can write it, I can use it. I just need more brain cycles to devote to learning C.

bsg75 · 12 days

Useful and consistent set of responses so far, thanks. I think I have my answer.

alllie · 15 days

Recall the name of the story? Sounds like an interesting read.

USS_MichelleBachman · 19 days

On an iPad?

bsg75 · 20 days

Python libraries are the killer feature.

Curious - what makes you a huge Lua fan?

bsg75 · 20 days

I would love to see the code! Such examples are very useful for learning the language.

My performance curiosity comes from the types of processes that draw in source data, and perform some sort of matching or iterating in memory before inserting into a database.

These cases are when an iterative approach has some advantage over native set based SQL.

ejrh · 20 days

Agreed. XML for data interchange = OK. For data storage = evil.

bsg75 · 20 days

Simple and pragmatic is good for me. I am also interested in performance, interested in testing some of my Python scripts converted to Lua and run on the JIT.

bsg75 · 23 days

IIRC, Oracle has the same terms. I think someone mentioned it in the article comments.

craig081785 · 23 days

My career has been a success mostly because of the attitudes (or associated ignorance) of developers like that guy. "Databases are hard. Lets build a website!"

I should buy him a beer.

bsg75 · 23 days

I too prefer PostgreSQL. There are cases where one of the big-$ platforms are needed, such as higher throughput requirements than I have: http://diznix.com/2010/07/27/postgresql-versus-oracle/

DB2 v10 has some interesting features, and I mostly keep an eye on it out of interest.

bsg75

TROPHY CASE

you'll need to login or register to do that

create a new account

login