Linux hot take: bash bashing 

The convention of having the command shell replace "*" with a list of all matching files in the current folder[1] only in is... well.

I understand how it's useful for really basic core file utilities.

For anything that needs to do recursive directory-searches, though, it really gets in the way and raises the bar for what the user has to know in order to make use of CLI in Linux.

I just now had a long conversation with an advanced bash user[2], and apparently there really is no way to get this information without setting an option in bash before running a command (and then presumably unsetting it afterwards, so as not to break other programs).

Just... why.

[1] ...and *only* the current folder... and only including folders that match the same pattern -- like "*.rb" would include a folder named "foldername.rb", which pretty much never happens

[2] Much thanks to sophia kara for hashing through this with me. I was very grumpy about it.

Linux hot take: bash bashing 

@woozle 1. What option is this?

2. Globbing is a convenience, but generally *Not Good Programming Practice*.

3. find | read or find | xargs is probably what you want. More specifically:

while find . <args> | read file do; echo ">>> $file <<<"; <processing on file>; done

I like to echo the name of the file(s) found, first, both as a verification of the find command/results, and as a progress indicator.

Linux hot take: bash bashing 

You're not afraid of filenames containing \newlines. Because you don't handle other people's filenames? For fun, create a file with a space in its name in your distro's package cache dir. Last time I tried that it completely stalled apt-get. sh is broken in many ways :^)

@dredmorbius @woozle

Linux hot take: bash bashing 

@RefurioAnachro I'm aware they can exist. For my purposes, that's not _generally_ a problem.

Though "find -print0 | xargs -0" generally addresses _that_ particular case.

(Wrapping the whole thing into a script is ... more work.)

I'm not sure precisely where that problem lies -- it's somewhere between the filesystem, perverse behaviour, and shell field/record delimiting conventions.


Linux hot take: bash bashing 

I'm not sure what to think of newlines in filenames. Or control characters. I like spaces in filenames, and I hate that android prohibits colons. I remember consciously deciding not to care about dos' limitations, then came android :-/

As nice as it can be, Unicode brings whole new classes of problems, normalization being one that can bit you even if (or especially when) you os has provisions for it.

@dredmorbius @woozle

Linux hot take: bash bashing 

But none of that is really perverse. This is: Did you know that not all linux syscalls check against '/' in filenames?

Anyways, I the classical hurdles mostly in conjunction with field splitting.

@dredmorbius @woozle

Linux hot take: bash bashing 

@RefurioAnachro @dredmorbius @woozle The whole problem is that UNIX (and even Plan 9) refuses to define any data structure encodings.
Could just use linear TSV and that would solve most problems. There is no reason to make intermediate data look like natural text. (and TSV is readable enough anyways)

Linux hot take: bash bashing 

Good point, @grainloom! Yes, I believe the os should provide standards for data exchange of all sorts. I really like how json simplifies things. I mean, at least for the js ecosystem. It may end up adding yet another variant and layer of complexity in other contexts.

@dredmorbius @woozle

Linux hot take: bash bashing 

I can haz 'description' field for files? What about user defined header fields? Of course, these would need to survive passing files between systems...

@grainloom @dredmorbius @woozle

@RefurioAnachro Files-with-metadata is actually a major component of a project I'm looking at.

Implementing that as a metadata-aware filesystem offers certain capabilities.

Though it also makes that portability / export issue a bit of a pain.

@grainloom @woozle

@dredmorbius @RefurioAnachro @grainloom

I very much want a means of entering per-file custom metadata. My current design for this involves an app, which could solve the portability problem by exporting data on a per-volume or per-folder basis.

@woozle @dredmorbius @RefurioAnachro not sure if that needs to be a whole new thing. could probably make something good enough with userspace file systems. much like how tag based file systems can still be represented as a directory tree.

@grainloom @dredmorbius @RefurioAnachro

I'm not sure I get what you have in mind.

I think the *main* metadata I want are hierarchical topic tags (the only piece that needs to be stored with the file would be a numeric ID) and a few timestamps (not necessarily the same as the file-creation or file-edited timestamps)... and of course a textual description. ...and there should be a facility for recursively searching files in a folder for metadata that matches a given criterion.

I'm not expecting anything much to happen with this unless I do it, given the current state of GUI file-searching tools.

@woozle @dredmorbius @RefurioAnachro grep-like search or indexed search?
what i was thinking of is just transforming the files into directories or something. it's mostly backwards compatible too. you can use it with tar, zip, etc. merging and diffing remains easily available.
i'll try to elaborate when i have more time.

@grainloom @dredmorbius @RefurioAnachro

Hmm... like, for myfile.jpg, you could have a .myfile.jpg/ folder (or some similar naming-scheme) with all the attributes as individual files underneath it...?

My programmer brain goes "agh, inefficient!" but I don't actually know how inefficient it would be. It would probably be a drop in the bucket.

Next step: need a GUI for managing all those meta-subfiles.

@woozle @dredmorbius @RefurioAnachro that's just a representation, just like how /dev is not really a file system.
the underlying data structure could be anything.


@grainloom @dredmorbius @RefurioAnachro

Ah, ok -- so this requires support within the OS or filesystem.

I'm looking for something that can work with existing OSs/filesystems/drives -- though if such a thing appeared in an OS, I'd still be interested in trying it out.

· · Web · 1 · 0 · 0

@woozle @dredmorbius @RefurioAnachro well, kinda, but on anything that supports FUSE (so, most relevant UNIX clones, AFAIK) this should work and would be able to interop with everything that uses files. i'm mostly sure that it also wouldn't require root to mount it. the underlying drivers don't really matter to it either.

@woozle @dredmorbius @RefurioAnachro if the target system supports stuff like mounthing SFTP shares, then it can do this too.

@grainloom @woozle @dredmorbius @RefurioAnachro git-annex supports both tags and key-value pairs. There's also xattrs, which are supported in various forms by Linux, the BSDs, Mac OS, and Windows. rsync supports them but I'm not sure if any other archivers besides tar do. Adding them to something like 7zip seems like it'd be easier than implementing a filesystem

Sign in to participate in the conversation

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!