This may make some people pull their hair out, but I’d love to hear some arguments. I’ve had the impression that people really don’t like bash, not from here, but just from people I’ve worked with.
There was a task at work where we wanted something that’ll run on a regular basis, and doesn’t do anything complex aside from reading from the database and sending the output to some web API. Pretty common these days.
I can’t think of a simpler scripting language to use than bash. Here are my reasons:
- Reading from the environment is easy, and so is falling back to some value; just do
${VAR:-fallback}; no need to write another if-statement to check for nullity. Wanna check if a variable’s set to something expected?if [[ <test goes here> ]]; then <handle>; fi - Reading from arguments is also straightforward; instead of a
import os; os.args[1]in Python, you just do$1. - Sending a file via HTTP as part of an
application/x-www-form-urlencodedrequest is super easy withcurl. In most programming languages, you’d have to manually open the file, read them into bytes, before putting it into your request for the http library that you need to import.curlalready does all that. - Need to read from a
curlresponse and it’s JSON? Reach forjq. - Instead of having to set up a connection object/instance to your database, give
sqlite,psql,duckdbor whichever cli db client a connection string with your query and be on your way. - Shipping is… fairly easy? Especially if docker is common in your infrastructure. Pull
Ubuntuordebianoralpine, install your dependencies through the package manager, and you’re good to go. If you stay within Linux and don’t have to deal with differences in bash and core utilities between different OSes (looking at you macOS), and assuming you tried to not to do anything too crazy and bring in necessary dependencies in the form of calling them, it should be fairly portable.
Sure, there can be security vulnerability concerns, but you’d still have to deal with the same problems with your Pythons your Rubies etc.
For most bash gotchas, shellcheck does a great job at warning you about them, and telling how to address those gotchas.
There are probably a bunch of other considerations but I can’t think of them off the top of my head, but I’ve addressed a bunch before.
So what’s the dealeo? What am I missing that may not actually be addressable?
I just don’t think bash is good for maintaining the code, debugging, growing the code over time, adding automated tests, or exception handling
If you need anything that complex and that it’s critical for, say, customers, or people doing things directly for customers, you probably shouldn’t use bash. Anything that needs to grow? Definitely not bash. I’m not saying bash is what you should use if you want it to grow into, say, a web server, but that it’s good enough for small tasks that you don’t expect to grow in complexity.
it’s (bash) good enough for small tasks that you don’t expect to grow in complexity.
I don’t think you’ll get a lot of disagreement on that, here. As mention elsewhere, my team prefers bash for simple use cases (and as their bash-hating boss, I support and agree with how and when they use bash.)
But a bunch of us draw the line at database access.
Any database is going to throw a lot of weird shit at the bash script.
So, to me, a bash script has grown to unacceptable complexity on the first day that it accesses a database.
We have dozens of bash scripts running table cleanups and maintenece tasks on the db. In the last 20 years these scripts where more stable than the database itself (oracle -> mysql -> postgres).
But in all fairness they just call the cliclient with the appropiate sql and check for the response code, generating a trap.
That’s a great point.
I post long enough responses already, so I didn’t want to get into resilience planning, but your example is a great highlight that there’s rarely hard and fast rules about what will work.
There certainly are use cases for bash calling database code that make sense.
I don’t actually worry much when it’s something where the first response to any issue is to run it again in 15 minutes.
It’s cases where we might need to do forensic analysis that bash plus SQL has caused me headaches.
Yeah, if it feels like a transaction would be helpful, at least go for pl/sql and save yourself some pain. Bash is for system maintenance, not for business logic.
Heck, I wrote a whole monitoring system for a telephony switch with nothing more than bash and awk and it worked better than the shit from the manufacturer, including writing to the isdn cards for mobile messaging. But I wouldn’t do that again if I have an alternative.
Bash is for system maintenance, not for business logic.
That is such a good guiding principle. I’m gonna borrow that.
“Use the best tool for the job, that the person doing the job is best at.” That’s my approach.
I will use bash or python dart or whatever the project uses.
I’m afraid your colleagues are completely right and you are wrong, but it sounds like you genuinely are curious so I’ll try to answer.
I think the fundamental thing you’re forgetting is robustness. Yes Bash is convenient for making something that works once, in the same way that duct tape is convenient for fixes that work for a bit. But for production use you want something reliable and robust that is going to work all the time.
I suspect you just haven’t used Bash enough to hit some of the many many footguns. Or maybe when you did hit them you thought “oops I made a mistake”, rather than “this is dumb; I wouldn’t have had this issue in a proper programming language”.
The main footguns are:
- Quoting. Trust me you’ve got this wrong even with
shellcheck. I have too. That’s not a criticism. It’s basically impossible to get quoting completely right in any vaguely complex Bash script. - Error handling. Sure you can
set -e, but then that breaks pipelines and conditionals, and you end up with really monstrous pipelines full ofpipefailnoise. It’s also extremely easy to forgetset -e. - General robustness. Bash silently does the wrong thing a lot.
instead of a
import os; os.args[1]in Python, you just do$1No. If it’s missing
$1will silently become an empty string.os.args[1]will throw an error. Much more robust.Sure, there can be security vulnerability concerns, but you’d still have to deal with the same problems with your Pythons your Rubies etc.
Absolutely not. Python is strongly typed, and even statically typed if you want. Light years ahead of Bash’s mess. Quoting is pretty easy to get right in Python.
I actually started keeping a list of bugs at work that were caused directly by people using Bash. I’ll dig it out tomorrow and give you some real world examples.
I honestly don’t care about being right or wrong. Our trade focuses on what works and what doesn’t and what can make things work reliably as we maintain them, if we even need to maintain them. I’m not proposing for bash to replace our web servers. And I certainly am not proposing that we can abandon robustness. What I am suggesting that we think about here, is that when you do not really need that robustness, for something that may perhaps live in your production system outside of user paths, perhaps something that you, your team, and the stakeholders of the particular project understand that the solution is temporary in nature, why would Bash not be sufficient?
I suspect you just haven’t used Bash enough to hit some of the many many footguns.
Wrong assumption. I’ve been writing Bash for 5-6 years now.
Maybe it’s the way I’ve been structuring my code, or the problems I’ve been solving with it, in the last few years after using
shellcheckandbash-language-serverthat I’ve not ran into issues where I get fucked over by quotes.But I can assure you that I know when to dip and just use a “proper programming language” while thinking that Bash wouldn’t cut it. You seem to have an image of me just being a “bash glorifier”, and I’m not sure if it’ll convince you (and I would encourage you to read my other replies if you aren’t), but I certainly don’t think bash should be used for everything.
No. If it’s missing
$1will silently become an empty string.os.args[1]will throw an error. Much more robust.You’ll probably hate this, but you can use
set -uto catch unassigned variables. You should also use fallbacks wherever sensible.Absolutely not. Python is strongly typed, and even statically typed if you want. Light years ahead of Bash’s mess. Quoting is pretty easy to get right in Python.
Not a good argument imo. It eliminates a good class of problems sure. But you can’t eliminate their dependence on shared libraries that many commands also use, and that’s what my point was about.
And I’m sure you can find a whole dictionary’s worth of cases where people shoot themselves in the foot with bash. I don’t deny that’s the case. Bash is not a good language where the programmer is guarded from shooting themselves in the foot as much as possible. The guardrails are loose, and it’s the script writer’s job to guard themselves against it. Is that good for an enterprise scenario, where you may either blow something up, drop a database table, lead to the lost of lives or jobs, etc? Absolutely not. Just want to copy some files around and maybe send it to an internal chat for regular reporting? I don’t see why not.
Bash is not your hammer to hit every possible nail out there. That’s not what I’m proposing at all.
And I certainly am not proposing that we can abandon robustness.
If you’re proposing Bash, then yes you are.
You’ll probably hate this, but you can use set -u to catch unassigned variables.
I actually didn’t know that, thanks for the hint! I am forced to use Bash occasionally due to misguided coworkers so this will help at least.
But you can’t eliminate their dependence on shared libraries that many commands also use, and that’s what my point was about.
Not sure what you mean here?
Just want to copy some files around and maybe send it to an internal chat for regular reporting? I don’t see why not.
Well if it’s just for a temporary hack and it doesn’t matter if it breaks then it’s probably fine. Not really what is implied by “production” though.
Also even in that situation I wouldn’t use it for two reasons:
- “Temporary small script” tends to smoothly morph into “10k line monstrosity that the entire system depends on” with no chance for rewrites. It’s best to start in a language that can cope with it.
- It isn’t really any nicer to use Bash over something like Deno. Like… I don’t know why you ever would, given the choice. When you take bug fixing into account Bash is going to be slower and more painful.
I’m going to downvote your comment based on that first quote reply, because I think that’s an extreme take that’s unwarranted. You’ve essentially dissed people who use it for CI/CD and suggested that their pipeline is not robust because of their choice of using Bash at all.
And judging by your second comment, I can see that you have very strong opinions against bash for reasons that I don’t find convincing, other than what seems to me like irrational hatred from being rather uninformed. It’s fine being uninformed, but I suggest you tame your opinions and expectations with that.
About shared libraries, many popular languages, Python being a pretty good example, do rely on these to get performance that would be really hard to get from their own interpreters / compilers, or if re-implementing it in the language would be pretty pointless given the existence of a shared library, which would be much better scrutinized, is audited, and is battle-tested. libcrypto is one example. Pandas depends on NumPy, which depends on, I believe, libblas and liblapack, both written in C, and I think one if not both of these offer a cli to get answers as well. libssh is depended upon by many programming languages with an ssh library (though there are also people who choose to implement their own libssh in their language of choice). Any vulnerabilities found in these shared libraries would affect all libraries that depend on them, regardless of the programming language you use.
If production only implies systems in a user’s path and not anything else about production data, then sure, my example is not production. That said though, I wouldn’t use bash for anything that’s in a user’s path. Those need to stay around, possible change frequently, and not go down. Bash is not your language for that and that’s fine. You’re attacking a strawman that you’ve constructed here though.
If your temporary small script morphs into a monster and you’re still using bash, bash isn’t at fault. You and your team are. You’ve all failed to anticipate that change and misunderstood the “temporary” nature of your script, and allowed your “temporary thing” to become permanent. That’s a management issue, not a language choice. You’ve moved that goalpost and failed to change your strategy to hit that goal.
You could use Deno, but then my point stands. You have to write a function to handle the case where an env var isn’t provided, that’s boilerplate. You have to get a library for, say, accessing contents in Azure or AWS, set that up, figure out how that api works, etc, while you could already do that with the awscli and probably already did it to check if you could get what you want. What’s the syntax for
mkdir? What’s it formkdir -p? What about other options? If you already use the terminal frequently, some of these are your basic bread and butter and you know them probably by heart. Unless you start doing that with Deno, you won’t reach the level of familiarity you can get with the shell (whichever shell you use ofc).And many argue against bash with regards to error handling. You don’t always need something that proper language has. You don’t always need to handle every possible error state differently, assuming you have multiple. Did it fail? Can you tolerate that failure? Yup? Good. No? Can you do something else to get what you want or make it tolerable? Yes? Good. No? Maybe you don’t want to use bash then.
You’ve essentially dissed people who use it for CI/CD and suggested that their pipeline is not robust because of their choice of using Bash at all.
Yes, because that is precisely the case. It’s not a personal attack, it’s just a fact that Bash is not robust.
You’re trying to argue that your cardboard bridge is perfectly robust and then getting offended that I don’t think you should let people drive over it.
About shared libraries, many popular languages, Python being a pretty good example, do rely on these to get performance that would be really hard to get from their own interpreters / compilers, or if re-implementing it in the language would be pretty pointless given the existence of a shared library, which would be much better scrutinized, is audited, and is battle-tested. libcrypto is one example. Pandas depends on NumPy, which depends on, I believe, libblas and liblapack, both written in C, and I think one if not both of these offer a cli to get answers as well. libssh is depended upon by many programming languages with an ssh library (though there are also people who choose to implement their own libssh in their language of choice). Any vulnerabilities found in these shared libraries would affect all libraries that depend on them, regardless of the programming language you use.
You mean “third party libraries” not “shared libraries”. But anyway, so what? I don’t see what that has to do with this conversation. Do your Bash scripts not use third party code? You can’t do a lot with pure Bash.
If your temporary small script morphs into a monster and you’re still using bash, bash isn’t at fault. You and your team are.
Well that’s why I don’t use Bash. I’m not blaming it for existing, I’m just saying it’s shit so I don’t use it.
You could use Deno, but then my point stands. You have to write a function to handle the case where an env var isn’t provided, that’s boilerplate.
Handling errors correctly is slightly more code (“boilerplate”) than letting everything break when something unexpected happens. I hope you aren’t trying to use that as a reason not to handle errors properly. In any case the extra boilerplate is…
Deno.env.get("FOO"). Wow.What’s the syntax for mkdir? What’s it for mkdir -p? What about other options?
await Deno.mkdir("foo"); await Deno.mkdir("foo", { recursive: true });What’s the syntax for a dictionary in Bash? What about a list of lists of strings?
- Quoting. Trust me you’ve got this wrong even with
Honestly, if a script grows to more than a few tens of lines I’m off to a different scripting language because I’ve written enough shell script to know that it’s hard to get right.
Shellcheck is great, but what’s greater is a language that doesn’t have as many gotchas from the get go.
One thing that I don’t think anyone else has mentioned is data structures. Bash does have arrays and hashmaps at least but I’ve found that working with them is significantly more awkward than in e.g. python. This is one of several reasons for why bash doesn’t scale up well, but sure for small enough scripts it can be fine (if you don’t care about windows)
I think I mentioned it, but inverse: The only data type I’m comfortable with in bash are simple string scalars; plus some simple integer handling I suppose. Once I have to think about stuff like
"${foo[@]}"and the like I feel like I should’ve switched languages already.Plus I rarely actually want arrays, it’s way more likely I want something in the shape of
@dataclass(frozen=True) class Foo: # … foos: set[Foo] = …That’s definitely worth mentioning indeed. Bash variables, aside from arrays and hashmaps that you get with
declare, are just strings. Any time you need to start capturing a group of data and do stuff with them, it’s a sign to move on. But there are many many times where that’s unnecessary.
A few responses for you:
- I deeply despise
bash(edit: this was hyperbole. I also deeply appreciate bash, as is appropriate for something that has made my life better for free!). That Linux shell defaults settled on it is an embarrassment to the entire open source community. (Edit: but Lexers and Parsers are hard! You don’t see me fixing it, so yes, I’ll give it a break. I still have to be discerning for production use, of course.) - Yes, Bash is good enough for production. It is the world’s current default shell. As long as we avoid it’s fancier features (which all suck for production use), a quick bash script is often the most reasonable choice.
- For the love of all that is holy, put your own personal phone number and no one else’s in the script, if you choose to use
bashto access a datatbase. There’s thousands of routine ways that database access can hiccup, and bash is suitable to help you diagnose approximately 0% of them. - If I found out a colleague had used bash for database access in a context that I would be expected to co-maintain, I would start by plotting their demise, and then talk myself down to having a severe conversation with them - after I changed it immediately to something else, in production, ignoring all change protocols. (Invoking emergency change protocols.)
Edit: I can’t even respond to the security concerns aspect of this. Choice of security tool affects the quality of protection. In this unfortunate analogy, Bash is “the pull out method”. Don’t do that anywhere that it matters, or anywhere that one can be fired for security violations.
(Edit 2: Others have mentioned invoking SQL DB cleanup scripts from bash. I have no problem with that. Letting bash or cron tell the DB and a static bit of SQL to do their usual thing has been fine for me, as well. The nightmare scenario I was imagining was
bashgathering various inputs to the SQL and then invoking them. I’ve had that pattern blow up in my face, and had a devil of a time putting together what went wrong. It also comes with security concerns, as bash is normally a completely trusted running environment, and database input often come from untrusted sources.)Could you explain those db connection hiccups you’ve seen?
Sure.
I’ll pick on
postgresbecause it’s popular. But I have found that most databases have a similar number of error codes.https://www.postgresql.org/docs/current/errcodes-appendix.html
It’s not an specific error that’s the issue, it’s the sheer variety of ways things can go wrong, combined with bash not having been architected with the database access use case in mind.
I find this argument somewhat weak. You are not going to run into the vast majority of those errors (in fact, some of them are not even errors, and you will probably never run into some of those errors as Postgres will not return them, eg some error codes from the sql standard). Many of them will only trigger if you do specific things: you started a transaction, you’ll have to handle the possible errors that comes with having a transaction.
There are lots of reasons to never use bash to connect to a db to do things. Here are a couple I think of that I think are fairly basic that some may think they can just do in bash.
- Write to more than 1 table.
- Write to a table that has triggers, knowing that you may get a trigger failure.
- Use transactions.
- Calling a stored procedure that will raise exceptions.
- Accepting user input to write that into a table.
One case that I think is fine to use bash and connect to a db is when all you need to do a
SELECT. You can test your statement in your db manager of choice, and bring that into bash. If you need input sanitization to filter results, stop, and use a language with a proper library. Otherwise, all the failure cases I can think of are: a) connection fails for whatever reason, in which case you don’t get your data, you get an exit code of 1, log to stderr, move on, b) your query failed cause of bad sql, in which case, well, go back to your dev loop, no?This is why I asked what sort of problems have you ran into before, assuming you haven’t been doing risky things with the connection. I’m sorry, but I must say that I’m fairly disappointed by your reply.
This is why I asked what sort of problems have you ran into before,
Lol. I’m fucking old. I don’t remember details.
assuming you haven’t been doing risky things with the connection.
Ha! Not a safe assumption, though. I’ve maintained even more shitty code than I’ve written, and that’s a lot! Lol.
I find this argument somewhat weak.
Lol. Me too. I was just trying to give the shorthand version.
Your explanation is much better.
Edit: but it doesn’t sound like you really needed a detailed answer from me, anyway.
I actually love listening to or reading someone else’s war story, and tbh the entire purpose of this post is to dig those up. Bash is one of those places where a lot about it is passed around as tribal knowledge. So I’d really love to hear how things have failed.
Fair enough.
Here’s what I remember: invoking
SQLcontaining inserts frombashhas resulted in lost data, when fairly unsurprising database things happened, sincebashdidn’t really expect to be in charge of logging the details of the attempted change. For the error, it wasn’t something surprising - maybe it was “max connections reached”, stuff that will just happen sometimes.The data loss was probably solveable in
bash, but the scripter didn’t think to (and probably would have needed more effort in a full development tool).Seems like something that can happen in any languages, though yeah, bash doesn’t make it easier, and it’ll depend on what the cli tool would return given the error (eg does it return some code in stdout or stderr, or some non-zero exit code). Depending on the library (in the language of choice), you may still have to handle such errors manually, eg adding the necessary logic to retry.
And in such a case, I guess it would be prudent to either make sure that the data can be retrieved again, or push it somewhere a bit more permanent (shared fs, or object storage), sort of in a dead-letter-esque style. Seems like the lesson here is to have a fall over plan. The failure mode is not something a proper language and library would necessarily help discover more easily though.
- I deeply despise
Just make certain the robustness issues of bash do not have security implications. Variable, shell, and path evalutions can have security issues depending on the situation.
Certainly so. The same applies to any languages we choose, no?
Bash is especially suseptable. Bash was intended to be used only in a secure environment including all the inputs and data that is processed and including all the proccess on the system containing the bash process in question for that matter. Bash and the shell have a large attack surface. This is not true for most other languages. It is also why SUID programs for example should never call the shell. Too many escape options.
Good point. It’s definitely something to keep in mind about. It’s pretty standard procedure to secure your environments and servers, wherever arbitrary code can be ran, lest they become grounds for malicious actors to use your resources for their own gains.
What could be a non-secure environment where you can run Bash be like? A server with an SSH port exposed to the Internet with just password authentication is one I can think of. Are there any others?
I was more thinking of the CGI script vunerability that showed up a few years ago. In that case data came from the web into the shell environment uncontrolled. So uncontrolled data processing where the input data crosses security boundaries is an issue kind of like a lot of the SQL injection attacks.
Another issue with the shell is that all proccesses on the system typically see all command line arguments. This includes any commands the shell script runs. So never specify things like keys or PII etc as command line arguments.
Then there is the general robustness issue. Shell scripts easy to write to run in a known environment and known inputs. Difficult to make general. So for fixed environment and known and controlled inputs that do not cross security boundaries probaby fine. Not that, probablay a big issue.
By the way, I love bash and shell scripts.
By the way, I would not consider logging in via ssh and running a bash script to be insecure in general.
However taking uncontrolled data from outside of that session and injecting it could well be insecure as the data is probably crossing an important security boundary.
At the level you’re describing it’s fine. Preferably use shellcheck and
set -euo pipefailto make it more normal.But once I have any of:
- nested control structures, or
- multiple functions, or
- have to think about handling anything else than simple strings that other programs manipulate (including thinking about bash arrays or IFS), or
- bash scoping,
- producing my own formatted logs at different log levels,
I’m on to Python or something else. It’s better to get off bash before you have to juggle complexity in it.
-e is great until there’s a command that you want to allow to fail in some scenario.
I know OP is talking about bash specifically but pipefail isn’t portable and I’m not always on a system with bash installed.
-e is great until there’s a command that you want to allow to fail in some scenario.
Yeah, I sometimes do
set +e do_stuff set -eIt’s sort of the bash equivalent of a
try { do_stuff() } catch { /* intentionally bare catch for any exception and error */ /* usually a noop, but you could try some stuff with if and $? */ }I know OP is talking about bash specifically but pipefail isn’t portable and I’m not always on a system with bash installed.
Yeah, I’m happy I don’t really have to deal with that. My worst-case is having to ship to some developer machines running macos which has bash from the stone ages, but I can still do stuff like rely on
[[rather than have to deal with[. I don’t have a particular fondness for usingbashas anything but a sort of config file (withexport SETTING1=...etc) and some light handling of other applications, but I have even less fondness for POSIXsh. At that point I’m liable to rewrite it in Python, or if that’s not availaible in a user-friendly manner either, build a small static binary.It’s nice to agree with someone on the Internet for once :)
Have a great day!
If you’re writing a lot of shell scripts and checking them with Shellcheck, and you’re still convinced that it’s totally safe… I tip my hat to you.
I agree with your points, except if the script ever needs maintaining by someone else’s they will curse you and if it gets much more complicated it can quickly become spaghetti. But I do have a fair number of bash scripts running on cron jobs, sometimes its simplicity is unbeatable!
Personally though the language I reach for when I need a script is Python with the click library, it handles arguments and is really easy to work with. If you want to keep python deps down you can also use the sh module to run system commands like they’re regular python, pretty handy
Those two libraries actually look pretty good, and seems like you can remove a lot of the boilerplate-y code you’d need to write without them. I will keep those in mind.
That said, I don’t necessarily agree that bash is bad from a maintainability standpoint. In a team where it’s not commonly used, yeah, nobody will like it, but that’s just the same as nobody would like it if I wrote in some language the team doesn’t already use? For really simple, well-defined tasks that you make really clear to stakeholders that complexity is just a burden for everyone, the code should be fairly simple and straightforward. If it ever needs to get complicated, then you should, for sure, ditch bash and go for a larger language.
That said, I don’t necessarily agree that bash is bad from a maintainability standpoint.
My team uses bash all the time, but we agree (internally as a team) that bash is bad from a maintainability perspective.
As with any tool we use, some of us are experts, and some are not. But the non-experts need tools that behave themselves on days when experts are out of office.
We find that bash does very well when each entire script has no need for branching logic, security controls, or error recovery.
So we use substantial amounts of bash in things like CI/CD pipelines.
Hell, I hate editing bash scripts I’ve written. The syntax just isn’t as easy
We are not taking about use of Bash in dev vs use Bash in production. This is imho incorrect question that skirts around the real problem in software development. We talk about use of Bash for simple enough tasks where code is rarely changed ( if not written once and thrown away ) and where every primitive language or DSL is ok, where when it comes to building of medium or complex size software systems where decomposition, complex data structures support, unit tests, error handling, concurrency, etc is a big of a deal - Bash really sucks because it does not allow one to deal with scaling challenges, by scaling I mean where you need rapidly change huge code base according changes of requirements and still maintain good quality of entire code. Bash is just not designed for that.
But not everything needs to scale, at least, if you don’t buy into the doctrine that everything has to be designed and written to live forever. If robust, scalable solutions is the nature of your work and there’s nothing else that can exist, then yeah, Bash likely have no place in that world. If you need any kind of handling more complicated than just getting an error and doing something else, then Bash is not it.
Just because Bash isn’t designed for something you want to do, doesn’t mean it sucks. It’s just not the right tool. Just because you don’t practice law, doesn’t mean you suck; you just don’t do law. You can say that you suck at law though.
If your company ever has >2 people, it will become a problem.
You’re speaking prophetically there and I simply do not agree with that prophecy.
If you and your team think you need to extend that bash script to do more, stop and consider writing it in some other languages. You’ve move the goalpost, so don’t expect that you can just build on your previous strategy and that it’ll work.
If your “problem” stems from “well your colleagues will not likely be able to read or write bash well enough”, well then just don’t write it in bash.
Yep. Like said - “We talk about use of Bash for simple enough tasks … where every primitive language or DSL is ok”, so Bash does not suck in general and I myself use it a lot in proper domains, but I just do not use it for tasks / domains with complexity ( in all senses, including, but not limited to team work ) growing over time …
In your own description you added a bunch of considerations, requirements of following specific practices, having specific knowledge, and a ton of environmental requirements.
For simple scripts or duck tape schedules all of that is fine. For anything else, I would be at least mindful if not skeptical of bash being a good tool for the job.
Bash is installed on all linux systems. I would not be very concerned about some dependencies like sqlite, if that is what you’re using. But very concerned about others, like jq, which is an additional tool and requirement where you or others will eventually struggle with diffuse dependencies or managing a managed environment.
Even if you query sqlite or whatever tool with the command line query tool, you have to be aware that getting a value like that into bash means you lose a lot of typing and structure information. That’s fine if you get only one or very few values. But I would have strong aversions when it goes beyond that.
You seem to be familiar with Bash syntax. But others may not be. It’s not a simple syntax to get into and intuitively understand without mistakes. There’s too many alternatives of if-ing and comparing values. It ends up as magic. In your example, if you read code, you may guess that
:-means fallback, but it’s not necessarily obvious. And certainly not other magic flags and operators.
As an anecdote, I guess the most complex thing I have done with Bash was scripting a deployment and starting test-runs onto a distributed system (and I think collecting results? I don’t remember). Bash was available and copying and starting processes via ssh was simple and robust enough. Notably, the scope and env requirements were very limited.
You seem to be familiar with Bash syntax. But others may not be.
If by this you mean that the Bash syntax for doing certain things is horrible and that it could be expressed more clearly in something else, then yes, I agree, otherwise I’m not sure this is a problem on the same level as others.
OP could pick any language and have the same problem. Except maybe Python, but even that strays into symbolic line noise once a project gets big enough.
Either way, comments can be helpful when strange constructs are used. There are comments in my own Bash scripts that say what a line is doing rather than just why precisely because of this.
But I think the main issue with Bash (and maybe other shells), is that it’s parsed and run line by line. There’s nothing like a full script syntax check before the script is run, which most other languages provide as a bare minimum.
OP could pick any language and have the same problem. Except maybe Python, but even that strays into symbolic line noise once a project gets big enough.
Personally, I don’t see python far off from bash. Decent for small scripts, bad for anything bigger. While not necessarily natively available, it’s readily available and more portable (Windows), and has a rich library ecosystem.
Personally, I dislike the indent syntax. And the various tooling and complexities don’t feel approachable or stable, and structuring not good.
But maybe that’s me. Many people seem to enjoy or reach for python even for complex systems.
More structured and stable programming languages do not have these issues.
As one other comment mentioned, unfamiliarity with a particular language isn’t restricted to just bash. I could say the same for someone who only dabbles in C being made to read through Python. What’s this
@decoratorthing? Or what’sf"Some string: {variable}"supposed to do, and how’s memory being allocated here? It’s a domain, and we aren’t expected to know every single domain out there.And your mention of losing typing and structure information is… ehh… somewhat of a weird argument. There are many cases where you don’t care about the contents of an output and only care about the process of spitting out that output being a success or failure, and that’s bread and butter in shell scripts. Need to move some files, either locally or over a network, bash is good for most cases. If you do need something just a teeny bit more, like whether some key string or pattern exists in the output, there’s grep. Need to do basic string replacements? sed or awk. Of course, all that depends on how familiar you or your teammates are with each of those tools. If nearly half the team are not, stop using bash right there and write it in something else the team’s familiar with, no questions there.
This is somewhat of an aside, but jq is actually pretty well-known and rather heavily relied upon at this point. Not to the point of say sqlite, but definitely more than, say, grep alternatives like ripgrep. I’ve seen it used quite often in deployment scripts, especially when interfaced with some system that replies with a json output, which seems like an increasingly common data format that’s available in shell scripting.
Yes, every unfamiliar language requires some learning. But I don’t think the bash syntax is particularly approachable.
I searched and picked the first result, but this seems to present what I mean pretty well https://unix.stackexchange.com/questions/248164/bash-if-syntax-confusion which doesn’t even include the alternative if parens https://stackoverflow.com/questions/12765340/difference-between-parentheses-and-brackets-in-bash-conditionals
I find other languages syntaxes much more approachable.
I also mentioned the magic variable expansion operators. https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
Most other languages are more expressive.
Your experiences are based on your familiarity with other languages. It may or may not apply to others. So to each their own I guess?
I do agree that the square bracket situation is not best though. But once you know it, you, well, know it. There’s also shellcheck to warn you of gotchas. Not the best to write in, but we have linters in most modern languages for a reason.
I actually like bash’s variable expansion. It’s very succinct (so easier to write and move onto your next thing) and handles many common cases. The handling is what I hope most stdlibs in languages would do with env vars by default, instead of having to write a whole function to do that handling. Falling back is very very commonly used in my experience.
There are cases where programming is an exercise of building something. Other times, it’s a language, and when we speak, we don’t necessarily want to think too much about syntax or grammar, and we’d even invent syntaxes to make what we have to say shorter and easier to say, so that we may speak at the speed of thought.
I don’t disagree with this, and honestly I would probably support just using bash like you said if I was in a team where this was suggested.
I think no matter how simple a task is there are always a few things people will eventually want to do with it:
- Reproduce it locally
- Run unit tests, integration tests, smoke tests, whatever tests
- Expand it to do more complex things or make it more dynamic
- Monitor it in tools like Datadog
If you have a whole project already written in Python, Go, Rust, Java, etc, then just writing more code in this project might be simpler, because all the tooling and methodology is already integrated. A script might not be so present for many developers who focus more on the code base, and as such out of sight out of mind sets in, and no one even knows about the script.
There is also the consideration that many people simply dislike bash since it’s an odd language and many feel it’s difficult to do simple things with it.
due to these reasons, although I would agree with making the script, I would also be inclined to have the script temporarily while another solution is being implemented.
I don’t necessarily agree that all simple tasks will lead to the need for a test suite to accommodate more complex requirements. If it does reach that point,
- Your simple bash script has and is already providing basic value.
- You can (and should) move onto a more robust language to do more complicated things and bring in a test suite, all while you have something functional and delivering value.
I also don’t agree that you can just solder on whatever small task you have to whatever systems you already have up and running. That’s how you make a Frankenstein. Someone at some point will have to come do something about your little section because it started breaking, or causing other things to break. It could be throwing error messages because somebody changed the underlying db schema. It could be calling and retrying a network call and due to, perhaps, poorly configured backoff strategy, you’re tripping up monitoring alerts.
That said, I do agree on it suitable for temporary tasks.
Wanna check if a variable’s set to something expected?
if [[ ]]; then ; fiHey, you can’t just leave out “test goes here”. That’s worst part by a long shot.
The rest of the syntax, I will have to look up every time I try to write it, but at least I can mostly guess what it does when reading. The test syntax on the other hand is just impossible to read without looking it up.I also don’t actually know how to look that up for the double brackets, so that’s fun. For the single bracket, it took me years to learn that that’s actually a command and you can do
man [to view the documentation.To be fair, you don’t always have to use the
[[syntax. I know I don’t, e.g. if I’m just looking for a command that returns 1 or 0, which happens quite a bit if you get to usegrep.That said,
man testis my friend.But I’ve also gotten so used to using it that I remember
-zand-nby heart :P
May I introduce you to rust script? Basically a wrapper to run rust scripts right from the command line. They can access the rust stdlib, crates, and so on, plus do error handling and much more.
Basically a wrapper to run rust scripts right from the command line.
Isn’t that just Python? :v
Yeah, sometimes I’ll use that just to have the sane control flow of Rust, while still performing most tasks via commands.
You can throw down a function like this to reduce the boilerplate for calling commands:
fn run(command: &str) { let status = Command::new("sh") .arg("-c") .arg(command) .status() .unwrap(); assert!(status.success()); }Then you can just write
run("echo 'hello world' > test.txt");to run your command.Defining
runis definitely the quick way to do it 👍 I’d love to have a proc macro that takes a bash like syntax e.gsomeCommand | readsStdin | processesStdIn > someFileand builds the necessary rust to use. xonsh does it using a superset of python, but I never really got into it.Wow, that’s exactly what I was looking for! Thanks dude.
How easily can you start parsing arguments and read env vars? Do people import clap and such to provide support for those sorts of needs?
I’d use clap, yeah. And env vars
std::env::var("MY_VAR")?You can of course start writing your own macro crate. I wouldn’t be surprised if someone already did write a proc macro crate that introduces its own syntax to make calling subprocesses easier. The shell is… your oyster 😜I can only imagine that macro crate being a nightmare to read and maintain given how macros are still insanely hard to debug last I heard (might be a few years ago now).
Check out @Ephera@lemmy.ml’s comment with existing libraries. Someone already did the work! 🎉
I’m so glad we have madlads in rust land xD Thanks for referring me to that!
proc macros can be called in tests and debugged. They aren’t that horrible, but can be tedious to work with. A good IDE makes it a lot easier though, that’s for sure.
As I’ve matured in my career, I write more and more bash. It is absolutely appropriate for production in the right scenarios. Just make sure the people who might have to maintain it in the future won’t come knocking down your door with torches and pitchforks…
That’s my take on the use of bash too. If it’s something that people think it’s worth bring their pitchforks out for, then it’s something you should probably not write in bash.