Redesign Aura

Create a design document in .md format for easy viewing on github. This also allows for code snippets to be added. The main section needed is Functional Requirements, as many of Aura's features were tacked on later because the use cases they involve were not known about at first. As many use cases as possible should be considered before recoding begins.

Related issues:

Hotedit: --hotedit install files. #208
Install method: support pkgbase() #213 aura -N #214 Support build directory that is different than the pkg cache directory #203 -M fails to take account of split packages #125 Interactive search/install #114 Aura's temporal directories doesn't work well with AUR packages that require manual downloads. #107 Color output with -S #202
Package signing: Aura cannot handle signed packages #211 Aura installation fails when Pacman SigLevel = PackageRequired #38
Help: --help for aura-only operations #111
Colour: Colours should only be shown when the output is a terminal #102 support pacman’s --color #158 Color output with -S #202
~~customizepkg: customizepkg failure & output #93 customizepkg is unsafe #143 Handling of customizepkg #198~~
Dependencies: Dependency check failure not verbose enough #215

Added skeleton for design doc - #217

@dhouck

There needs to discussion about --hotedit and customizepkg. @dhouck I merged your pull request, and things looked good, but when I got thinking about it, and read this I think forgoing any PKGBUILD editing might be the way to go.

The Problem

Aura's current behaviour is this, assuming --hotedit:
0. Call API to confirm existence of package.

Download PKGBUILD.
Write to a temporary file.
Open it in editor.
Read it back as data.
Download package sources (PKGBUILD on AUR, .install, etc.)
Overwrite that PKGBUILD will the edited one.
Check dependencies
Call makepkg

With AUR 3.0, dependency information is going to be available via the AUR API, meaning the PKGBUILD will never have to be downloaded in first place. Fewer HTTP calls mean a faster Aura. The process becomes:
0. Call API to confirm existence of package and get its dep info.

Check dependencies.
Download package sources (PKGBUILD on AUR, .install, etc.)
Call makepkg.

As dep checks happen before the PKGBUILD is downloaded, editing the dep fields in the PKGBUILD is pointless. If the user were given the opportunity to alter the deps given by the API, they would also have to alter the PKGBUILD in the same way, or else makepkg would crash.

The Case for `--(hot)edit`

Some packages require you choose a certain build method by (un)commenting certain sections in build(). xmobar-git is one such example. Which is better:

Allow the user to use --edit to change the PKGBUILD so they can choose their build style, or;
Disable all manual editing and tell them to build it manually.

I would still like a --edit.

@bb010g

@bb010g Which would allow the user to edit all files in the downloaded tarball? They would have to understand that editing the deps fields wouldn't do anything.

Just have it edit the metadata from the AUR. (That could go well with something like customizepkg built in, but acting on those fields.)

I'll have to see what the shape of the metadata is before I can make a call on that.

For the ABS, wouldn't you still need to read the PKGBUILD to find the makedepends and checkdepends (even if you use the local binary package database for the regular and optdepends)? Given that, would it really be simpler to use the AUR's dependency list instead of reusing the dependency checking code needed for the ABS?

Also, for the current (pre-AUR3) workflow, what's the point of doing steps 1 and 5 separately? It seems like it might be more efficient to do the following, but if this approach is missing something crucial:

Call API to confirm existence of package.
Download package sources (PKGBUILD on AUR, .install, etc.)
Prompt for edits; open relevant files in editor
Check dependencies
Call `makepkg`

EDIT: Start list at 0 to match ones above

Question: How much dependency information is stored in the database? Are {make,check}depends excluded?

As for the current flow, you're probably right that doing those things separately is silly. There are redundant http requests, and the size difference between downloading a .tar.gz and just the PKGBUILD is nearly negligible.
The only problem I can see is getting back into a folder once you've untarred it. That is,
with your suggested flow, each tar would be downloaded, opened, and have their PKGBUILD read. Then aura has to back out of that folder, go into another one, and after all dep checks are finalized, go back into the folders in the proper order and call makepkg in there.
So the problem is, right now all tars are downloaded into temp folders that aura's haskell underpinnings delete automatically after building. This gives the following sort of folder hierarchy:

foo1234            # temp name created by haskell
  |--- foo.tar.gz  # the downloaded tarball
  |--- foo/        # the tarball's uncompressed contents
        |--- PKGBUILD
        |--- foo.install

And every folder for every package would look like this. Navigating from folder to folder would take some regexing on package names to find out where to navigate to. Right now, it just does it one at a time. The pre-downloaded PKGBUILD is held internally, dep checks are done, packages are sorted into a nice build order, and then one-by-one their tars are downloaded and built.

Regardless, we don't have to worry about any of this in Aura 2 because of the new API coming, and the libalpm bindings to rely on. I agree with you that the current method is inefficient, but until Aura 2 it probably won't be changed.

I assume the question at the top of your comment was about libalpm, because AUR 3.0 handles all four types of dependencies and even allows separate dependencies in split packages (which I don't think Aura correctly handles now).

I believe ALPM only records regular and optdepends (which makes sense, given that it's designed for binary packages and therefore doesn't need to know about the others). The ALPM doxygen documentation only gives functions to retrieve these two dependencies (or it puts the makedepends and checkdepends functions somewhere other than where it puts the depends and optdepends ones. Thus, unless it's hiding somewhere I (and Google) haven't seen it, it's undocumented, or you want to drop ABS support entirely, you'll need to get these fields from PKGBUILDs directly when using the ABS.

Because of the ABS, the updated workflow you posted might not always be usable and some variant of the existing one might be necessary, at least sometimes, so I have another question about it. Would there be a problem with having a single temp directory with all the packages under it, like below:

Aura1234            # temp name created by haskell
  |--- foo.tar.gz  # the downloaded tarball
  |--- foo/        # the tarball's uncompressed contents
        |--- PKGBUILD
        |--- foo.install
  |--- depend.tar.gz # foo depends on depend
  |--- depend/
        |--- PKGBUILD
        |--- depend.rc

Just a few remarks, since I had also time to think about these issues for another project:

Support for customizepkg is dropped, as AUR 3.0 provides dependency information via its API.
Support for --hotedit dropped. As dependency checking is done through the API and not the PKGBUILDs, editing PKGBUILDs could cause a discrepancy between aura and makepkg. If you need fine-grained control over how the package is being built, building it manually with makepkg is recommended. If you're trying to fix a broken package, please contact the maintainer.

I fully agree with the above. There shouldn't be any reason to modify dependencies in the PKGBUILDs in the first place. And if there is one, an additional PKGBUILD should probably be uploaded in the AUR instead of doing "magic" locally (that's the very reason I believe the idea behind customizepkg is silly in the first place).

An --edit flag should be kept for special cases like xmobar-git, although any modification to the deps fields would be ignored.

Just have it edit the metadata from the AUR.

The .AURINFO metadata is only useful to make the complete info available in the RPC interface in a secure way (and adding split package support in the AUR too), so "editing the metadata" would be the equivalent of editing the Json response.

Then, it's possible to edit the json info on the fly, if you keep one info call per package or dependency. But Aura should use the multiinfo RPC call instead, because it is way faster as it requires only one call per level of dependencies, no matter what is the total number of dependencies in the transaction.

Of course it would also theoretically be possible to edit the returned multiinfo answer, but that's would probably a nightmare for the user in practice. Better forget the customizationpkg thing entirely, it's not worth it (and silly in the first place imho).

or you want to drop ABS support entirely, you'll need to get these fields from PKGBUILDs directly when using the ABS.

I know some core pacman developers are interested in integrating ABS closely with pacman itself (read it some time ago on the ML). I haven't seen any patch for it yet, but I know this will eventually happen. Maybe in two weeks, maybe in two years, I don't know. So it's up to the aura dev to decide if it is worth it integrating it now, or if the extra work and time would be better invested in some other area of aura.

Premise: If you know enough about package building to know you want different dependencies, you're smart enough to build them manually while hand-editing the PKGBUILD.
Conclusion: No dependency editing will be supported. If dependencies are broken, the maintainers need to be contacted and the AUR versions updated. Things like --hotedit are actually letting packages lag behind if users are able to edit everything themselves on the fly, because it doesn't force the maintainers to keep up to date packages. And if the users really want custom deps that badly, they can building manually as stated above.

An --edit flag will be used purely for editing build behaviour. Any dep changes will be ignored by Aura (but notably, not by makepkg).

And yes, one multiinfo call for everything will be quite nice.

Since you've decided to keep hotediting, is there a reason not to also keep customizepkg support? If it's made clear that customizepkg also cannot be used for Aura's metainformation useful for changing building behavior when you want to make the same changes to a new version's files. Often, you want to keep the same changes whenever you reinstall the package. With customizepkg (or at least customizepkg-scripting or customizepkg-patching; I don't know about the original), this can be handled automatically; without, you need to re-run Aura with --hotedit (and -k, but I always use that) in order to see exactly what changes you made and to make them again.

I'll agree that changing dependencies isn't that useful, but there is a case where "contact the maintainer" or "release a clone package" aren't better options: adding optional dependencies to the depends array (because you want the dependency, and installing optional dependencies still doesn't work well whether or not you use --asdeps even if things are getting better). In this case, it's possible to install the optdepend --asdeps manually, then to make the changes to the package and just not worry about Aura not knowing. This would be a bit hacky, but would still work even without Aura tracking package metadata changes.

Often, you want to keep the same changes whenever you reinstall the package.

What prevents you from uploading your own version in the AUR? I don't see in which way that wouldn't be a better option than patching again and again a package locally.

In general: when somebody wants to make a change to an package and they believe their change would only benefit a small number of people (with others neutral or disliking the change), it is generally not worth uploading the package to the AUR. It's a minor hassle for the maintainer (and doesn't get rid of the problem of "patching again and again a package locally"), and if only a small number of people would prefer their version to a version already in the AUR or the official repositories, it doesn't give much benefit. Often, this change may only be useful to that one person (one example might be somebody who wants to configure dwm but still have it managed by a package manager, but I'm not sure if this example works because I don't personally use dwm or any of the other several tiling window managers that require recompilation for a configuration change ).

The specific package that prompted my earlier comment: There's a package in the community repo called terminus-font. There are also four versions of it in the AUR with different configurations, only one of which has much support and none of which are up-to-date. I want a fifth configuration which is almost but not quite identical to one of the ones without much support already in the AUR, so uploading the changed package to the AUR (and then maintaining it) probably wouldn't be worth the effort and might make things harder for anybody looking through the AUR (because they now have more nearly identical choices and they're unlikely to want my particular changes).

Could you not just keep a local PKGBUILD that you bump the version of every once and a while?
customizepkg has in the past introduced a number of problems (which to use? how to catch its errors? is it secure at all?) Making aura handle customizepkg properly is probably much more work than the payoff would be.

You mention avoiding confusing AUR users with multiple versions on the AUR. I agree with this. We've arrived at one of the fundamental problems of the AUR: it's filled with garbage. Vast expanses of 0-vote, out-of-date, orphaned packages. Is there any system in place to have these removed?

In the mean time, aura -A{i,s} show maintenance status and votes. That's about the only tool we have right now for telling what packages are legitimate.

@atondwal

@atondwal I didn't know there were non-Linux versions of Arch. Provided they have pacman and makepkg installed, then yeah, aura should work unmodified.

Maybe we could have names like Distribution.Arch and Distribution.Debian for pacman/dpkg stuff respectively?

Or take an Aura topname?

I'd be for just Aura.

Where a hook library would be: Aura.DistroName.

+1

#332 brought up an interesting point. Currently, pacman uses both Q and D operations for local database operations (queries & changes) and R for local package removal, and uses S for remote package addition and database queries, T for a mix of remote and local queries (dependency testing for wanted remote packages on the local system), and U for local package addition. V is used for version testing. We'll have to find some way to unify all these into top level operations. I'd suggest a model of two operations per hook: e.g. for the main system package manager, S for remote and Q for local. pacman's R, D, & U would then become sub-operations of Q and pacman's T would become a sub-operation of S. I'd imagine v (née V) would become like h and give info on the parent hook, leaving V for global aura version info.

My propositions for renaming top-level ops is covered in the Doc. I like the idea of relegating the other ones to subops of the top-level ones.

The main issue with something like a global R operation is that you can have different packages from different hooks that have the same name. What happens then? An interactive disambiguation would be my first thought, but you can't easily script that. My next thought is a sort of namespacing, and that is what the two operations per hook pattern would provide. Also, on the topic of scripting, it would be nice to have a long, unique variant of each operator (as to provide consistency across setups in scripts). E.g. on an Arch system, -S would also be --alpm and -A would be --aur. A is unique here, as it depends on -L (--alpm-local) for local package management and thus is an exception to the two operations per hook pattern. However, something like --linuxbrew would need a --linuxbrew-local.

I have been flirting with the idea of multiple Hook sets on one machine (which you seem to be implying here). As pacman does now with haskell packages, could it be the domain of Aura to interface to all package managers on a user's system?

Then they make calls like:

aura -S firefox <- the default, with no git-like command.
aura cabal -{S,L} lens <- Still install/query the same way.
aura npm -{S,L} grunt
aura pip -{S,L} django

Some general -Ss function could search through all Hook sets and provide multiple sources for the same package.

aura -Ss lens then gives:

haskell-core/haskell-lens 1.x.x: blah blah blah
cabal/lens 1.x.x: blah blah blah

@fosskers

@fosskers that would probably make the day of people who use all these different package manager and would be very useful.

Is it easier for someone to do aura cabal -S lens or cabal install lens? I'm thinking the later. That said, I think this functionality would really shine during querying.

Output like aura -Ss lens above with local packages would also be very convenient:

> aura -Ls lens
haskell-core/haskell-lens 1.x.x: blah blah blah
cabal/lens 1.x.x: blah blah blah

So that you can see what packages, from any manager, are installed on your system.

Relevant conversation on the Gitter on July 27th and 28th. The direction is shifting away from "support _ALL_ the things" to "support ALPM and the AUR well".

Design then may be informed by a proper port of libalpm.

Navigation Menu

Repository navigation

Redesign Aura #217

The Problem

The Case for `--(hot)edit`

140 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Redesign Aura #217

Description

Activity

fosskers commented on May 14, 2014

The Problem

The Case for --(hot)edit

bb010g commented on May 14, 2014

fosskers commented on May 14, 2014

bb010g commented on May 14, 2014

fosskers commented on May 15, 2014

dhouck commented on May 15, 2014

fosskers commented on May 15, 2014

dhouck commented on May 15, 2014

rmarquis commented on May 15, 2014

fosskers commented on May 15, 2014

dhouck commented on May 16, 2014

rmarquis commented on May 16, 2014

dhouck commented on May 16, 2014

fosskers commented on May 16, 2014

140 remaining items

fosskers commented on Aug 21, 2015

atondwal commented on Aug 21, 2015

fosskers commented on Aug 22, 2015

bb010g commented on Aug 24, 2015

fosskers commented on Aug 27, 2015

flamusdiu commented on Aug 27, 2015

bb010g commented on Sep 2, 2015

fosskers commented on Sep 3, 2015

bb010g commented on Sep 3, 2015

fosskers commented on Sep 3, 2015

flamusdiu commented on Sep 4, 2015

fosskers commented on Sep 8, 2015

bb010g commented on Aug 4, 2016

fosskers commented on Aug 4, 2016

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

The Case for `--(hot)edit`