Introducing Shaman, a new universal package management frontend
Hello people,
yes, you can shiver at the title: we have (are still, actually) done a new package management frontend. Why? Because KDE missed a great one. Maybe some of you using Arch Linux will recognize the name Shaman: it is the default package manager frontend in Chakra, and my very first project with Qt. As such, it has grown quite old, and it’s showing the limits of my early programming skills, which made it almost unmaintainable by now. So me and Lukas (boom1992) decided to take it on a whirl and rewriting it from the ground up, with a more ambitious goal. We had some clear goals in mind:
- It should have been extensible to any package management system and integrated with it
- It should remain simple, and be extendable to the sky throughout plugins
- It should be completely asynchronous to bring the user the best possible experience
- It should be scriptable, to lower the barrier of the contributors and open up more possibilities to backends not in the C realm
There is one platform which actually makes this all possible, and it’s KDE π So we created Shaman 2. What is it?
Chapter 1: Shaman for Developers
Before being a package manager, it is a library. Think about it as the Solid or the Phonon of package management. This library provides an easy to use interface to implement frontends, and an abstraction layer for building plugins and backends. A backend is an interface to a package management system, such as apt, yum, packagekit, alpm, whatever. It is implemented as a static plugin that will be chosen upon build. People who need to implement a backend have to reimplement Shaman::BackendInterface and Shaman::Package, and make it work. It’s really easy to do: the current packagekit backend weights ~900 lines of code (and a lot of them are made of strings) and it’s almost feature complete.
Also, in the spirit of giving the backend developers the maximum flexibility, not forcing them into workarounds and getting the best possible performance/user experience ratio, you can implement delayed loading, custom caching of the information, custom progress reporting (soon), custom searching implementations and more. Shaman comes already preloaded with “Insane rockstar settings” (cit.), a backend for PackageKit and one for Aqpm (pacman), but we are looking forward to people who would like to port Shaman 100% natively on platforms not fully supported by PackageKit. For example, Shaman will support soon streaming transaction questions with custom widgets and creating scripted backends. Did you see any references to apt and/or debconf here? I did not.
At the moment all the code in Shaman runs unprivileged: in the near future, though, it will be given the possibility to create a backend with root privileges just by giving a switch to Shaman at compile time. Thanks to some PolicyKit+DBus magic, the backend (and only the backend) will be run privileged and will require authorization to perform any actions. You still have the option of running Shaman as root, but these days it’s really nonsense. You can also simply bypass such a possiblity and implement this straight into the backend (as Aqpm and PackageKit do), but this boils down to be your choice.
What is extremely cool about Shaman, though, is its plugin system. Almost any functionality can be implemented through plugin, from the GUI or the core side. In fact, the 2 main widgets of the Shaman interface are plugins as well. Plugins have a very important role in Shaman, and can also play an active part in transactions. Every plugin, in fact, can register some “Hook”, that can be run after and/or before a transaction. Pre transaction hooks have also the power of reporting fatal errors and aborting the upcoming transaction before it even starts. As you can imagine, there’s a lot of room for making Shaman fully integrated with the underlying distribution and perform some annoying tasks not belonging to the package management system. For example, what about a plugin that after a transaction checks if the kernel was upgraded, and in case rebuilds the VirtualBox kernel module?
Scripting support, as I told you, is a priority. Even if it’s not working right now due to constant changes in the API, it will be made ready as soon as we reach a semi-stable API, and it will use Kross. The basics are already there and ready, so it’s just a matter of time. Help welcome on this side, especially from some Kross masters around π
The API is still unstable but it’s stabilizing as I write this. You know where to reach me, if you’re interested, we can have a chat and you can bug me with some questions π I look forward to plugin and backend developers
Chapter 2: Shaman for users
Anyway, what about the user interface? The user interface is meant to be non-modal when and where possible, simple & easy to use yet detailed, still remaining beautiful. How does it look like as of today? (Some screenshot here on Arch, some on Fedora, using Aqpm and PackageKit)
-
- Shaman as it appears on Arch now
-
- Shaman is loading itself
-
- Shaman on Fedora waiting for a lock
-
- Shaman on Fedora
-
- Package download
-
- After download, queue processing takes place
-
- Transaction completed
P.S.: The interface is changing by the minute and the default dialog sizes are much smaller than shown. Hey, this is a preview footage after all π
You might or might not have noticed some small differences in the PackageKit and Aqpm UI. This is because Shaman adapts its UI depending on the capabilities of the backend. So you will never be shown something your backend does not know about. Well, unless the backend developer is cheating, of course π
Shaman implements also some neat stuff: for example, your queue is retained on exit. This means that if you mark some packages to be processed, then your girlfriend calls you and you realize being already 30 minutes late for your date, just don’t worry: close Shaman and turn off your PC: the next time you will start Shaman, it will automagically restore the last queue, which will be ready to be processed. There is also a transaction logger ready for you, so that you won’t lose any bit of history.
There is also security: Shaman is meant to be run as a standard user, using technologies such as PolicyKit and KAuth for granting authorizations for privileged actions.
And it installs, removes and upgrades your packages too!
Near future development
Shaman as the frontend is only the first product in our quest to provide a fully integrated and KDE branded experience for package management. Those of you who used/tried Chakra might have run into, and hopefully loved, Chase, the automatic update system. Chase will be improved and ported over to libshaman (not a lot of work actually), to provide automatic updates in a fully integrated way. And there is also some stuff boiling in our minds to bring to Plasma some of the package management: we are planning on a Plasmoid which will serve as a global visualizator for transactions done through libshaman, and will also provide some very basic package management functions. All the three tools (Automatic updates, plasma widget/engine, Shaman) will be tightly integrated and cooperating, instead of conflicting (just as it happens with Shaman 1 and Chase in Chakra right now), to provide users an easy and integrated way of managing their software.
That’s all, folks
And I sincerely hope I raised some interest π I am actually looking forward to distribution developers to get excited and use Shaman in their own distros. Up to now I have a VM for Fedora and Debian where Shaman runs great using the PackageKit backend.
The most important thing, is that Shaman is still not ready to be used. But for the brave souls and curious developers, playground/sysadmin/shaman is the place to be
A Quiet Place 
This is what Kubuntu needs.
Fantastic news! I can’t wait to see Shaman working in Debian, so I can get rid of Synaptic and rest of GTK.
Cool – now I not only have to manage the packages for my system, but also the scripts for my package manager.
Plus, either you totally failed to understand what PackageKit is, or you just don’t care – otherwise you would at least have mentionned why using PackageKit directly is no option.
@Anonymous: Either you totally failed to read the post and find the lines in which is explained what Shaman is and the reason why one might want to substitute packagekit in certain cases (and given that I wrote a full backend for PackageKit I’m quite sure of what I’m talking about, I don’t know if this the same for you), or you’re simply trolling around.
If I didn’t care about PackageKit I wouldn’t have written a backend for it, by the way
Psssh, Linux users don’t have girlfriends.
That made my day π
Hy. Would it be possible, with shaman, to get parrallel queu management? I mean, exept for updates, I usualy use my package manager for a large bulk of installs, an always forget a few packages. having to wait for a transaction to acheive before going on to other ting is anoying. Like, you whant to install, say, krita for testing, but you realise you need it just after you launched a big update. So you switch to cooking, and when you get back to your computer, you forget you need to install krita untill next update. And it took me 3 updates before I managed to install krita. If I could have just added Krita to the queu wile downloading/installing updates, or working on a second queu, it would have been a big annoyance gonne.
So… What are the dependencies of shaman ? KDE runtime / KDE Libs / Qt only ??? Could lib shaman be used on a plateform without X so that one can implement a concurent package management atop of it?
Thanks for the work anyway. Recently switch to chakra wich roks
I was going to ask the exact same thing.
Unfortunately is impossible – Shaman is just a layer, not an underlying system. For example, you would have to implement such a thing in Yum, then Shaman would be able to use it. It’s mostly like wanting to write a driver for Solid, just to clarify π
i think what they mean is to enqueue krita to be installed after the updates are finished, so basically make the interface non-modal to allow edits while transactions take place and a queue to hold future transactions that will take place when the current ones are finished. so I don’t think yum needs to even know about this, just let it finish the current updates and then tell it you want to install krita (as if the user clicked it after the updates finished).
Ah-ah, then I misunderstood. The only burden I see is dependency computation but… something to think about π I just can tell I will look into it
Yes, what vespas said is exactly what I’m talking about. This is the feature I’ve been begging from Ubuntu / KDE / PackageKit people for ages. All have basically said it would be someone else’s responsibility.
An interesting side effect of this kind of system, would likely be fixing “dpkg lockout” for new users who had their packagemanger crash.
Hey man, that’s fantastic! I was really looking forward to see Shaman2.
Ps.: Don’t forget about updating the KDE Techbase site with KAuth info, you said you would do that ASAP π
You’re right – sorry π¦ I’ll find some time for it in the next days. In the meanwhile a decent and quite extended tutorial is here: http://api.kde.org/4.x-api/kdelibs-apidocs/kdecore/html/namespaceKAuth.html
Hmmm, how broadly are we talking about here on terms of “package managers” …could Portage (Gentoo) or Ports (*BSD) work with it?
I don’t really know extremely well how they work, especially Ports. However Shaman abstraction is very wide. A backend is requested to expose:
– A list of packages with some details about them
– A method to process these packages
– A way to report a progress on a transaction
If they are able to provide such information, then it should be fine to use them with Shaman, so I suppose the answer is yes. It would be a nice experiment using it as a frontend to Portage, given that you can implement additional tasks and configurations through plugins. If you are interested in implementing a backend for them it would be more than awesome
Sadly, I’m not much (=understatement!) of a coder, but if this was the case, I could at least give some Gentoo devs a hint that Shaman could be a worthy successor of the now dead Kuroo.
The problem with Portage (and Ports) is that it compiles software, so the “packages” are nothing more then scripts that tell Portage how to compile the software. Also there’s variables like the USE flags, with which you can turn on or off certain support (e.g. “-gtk +qt” would disable the GTK interface and enable the QT interface of a certain program) globally or per-package.
I see. Well, “Packages” again is an abstract concept, which translates to “an entity that is able to provide information about a software on the system”, so it doesn’t matter if it’s virtual or real, as long as it provides information.
For the USE flags, you would have the solution by using a plugin for configuring the action inside a package. As you have read in the post, the package class is reimplemented by the backend so you can do anything inside it. I will add support for adding custom data to a package (thanks for making the idea come to my mind!), afterwards you can simply hook in a GUI plugin that sets some data on the current package (already possible), and there you have it.
Feel free to point any interested developers to me, if they’re interested.
Hmm, it sounds like it could be done. When I’ll find some time I’ll definitely try to get some of the Gentoo devs to take a look at it and point them your way.
Gentoo’s Portage was lacking a KDE GUI for too long! XD
(and when/if Portage can use Shaman, it shouldn’t be too much work to get *BSD’s Ports working as well!)
Some things that come to mind:
– emerge (the default Portage package manager, there is also Paludis and pkgcore) does not report progress information during the installation/removal of a package. Since installing a package can take hours (openoffice built from source, for example), that’s definitely an issue.
– If there are problems during the dependency resolution, the user should get all information emerge provides (which is a lot, sometimes) – like which package blocks which, what can be done to resolve the conflict (like changing a USE flag), etc. – I don’t know if/how this information can be obtained from emerge in a structured way.
– handling of package-specific USE flags (and other things, like setting KEYWORDS, MASKing or UNMASKing packages, etc.) would have to be done via the dedicated files in /etc/portage
– handling of global USE flags and other things like MAKEOPTS etc. would have to be done in /etc/make.conf – but maybe that is out of the scope of Shaman…
Point to point:
– It’s not a big issue, since you are able to provide the detailed progress line per line in the details window – which is what you get with emerge right now. If you wanted to, you could even write the terminal output line-per-line. You would just stream the package progress to 100% when it’s done, and that’s it.
– I don’t know if emerge provides such information, I can only say that Shaman has now a mean for displaying them in a decent way (WidgetFactory)
– (Applies also to the next point) Each backend can have its own configuration module(s) in which you can do whatever you like. For flags in /etc/portage you could either use a configuration module (not so great), or add a custom widget to package information (yes, you can do that) in which you’d put the logic for changing package flags and stuff. For /etc/make.conf, it’s just a matter of parsing it and exposing it in the backend configuration. Of course you already have the tools (KAuth) to handle privileged actions, so you can definitely modify those files without issues.
Interesting, a package management abstraction over another package management abstraction (aka PackageKit).
Besides PackageKit has already Qt wrappers and a KDE GUI (KpackageKit)
I agree with anonymous, you did miss the point about PackageKit.
PK is NOT a package manager, it is an abstraction over package managers (it has yum, apt, conary etc … backends) like Shaman. PK already uses PolicyKit for privileges management, it already expose a D-Bus API, it exposes API for applications (for instance, it is used to download missing plugins, codecs, fonts or even command line tools), it has plugins. It even integrates with NetworkManager, like blocking updates when using a mobile broadband connection.
What shaman does that PackageKit doesn’t ? I think you should contact PackageKit people on FreeDesktop.
http://www.packagekit.org/pk-intro.html
>Interesting, a package management abstraction over another package >management abstraction (aka PackageKit).
>Besides PackageKit has already Qt wrappers
Which I used for Shaman, just by chance
> and a KDE GUI (KpackageKit)
Which is completely different from Shaman
>I agree with anonymous, you did miss the point about PackageKit.
>PK is NOT a package manager, it is an abstraction over package managers (it has >yum, apt, conary etc β¦ backends) like Shaman.
Uff… I know. Why are you people assuming I don’t, I wonder? Better getting some real world information first maybe? It exposes, however, a package manager-like interface, something you gurus should know about, so from an user (and this includes a developer implementing it into an application) it still remains a package manager.
> PK already uses PolicyKit for privileges management, it already expose a D-Bus >API, it exposes API for applications (for instance, it is used to download missing >plugins, codecs, fonts or even command line tools), it has plugins. It even >integrates with NetworkManager, like blocking updates when using a mobile >broadband connection.
I know as well. I’ve written about it in the post as well.
>What shaman does that PackageKit doesnβt ?
Something more and something less. Shaman is designed to be used as a GUI interface and it’s a partial abstraction as the backend is static whereas you can compile more than one backend into PK. It is mainly designed to be an interface to PackageKit, BUT, as you should know since it looks like I know nothing about PK and you people do, PackageKit is not a complete abstraction layer.
PackageKit impedes (in my opinion, correctly) user interaction during a transaction, something apt needs (that’s the reason of my reference). PackageKit has a precise modus operandi that unfortunately cannot suit some backends. Shaman instead focuses on letting total flexibility, so it’s meant to be used with PackageKit where PackageKit rocks (like Fedora) or with a native backend where packagekit doesn’t rock and probably will never do (Arch)
what kollum suggests would be great to have!
(shaman 1 _is_ great to have also btw, thank you)
@Karl: Sometimes it is just easier to start a new project and let it compete with others. Seriously, if you have a look at KPackageKit, neither does the GUI rock nor does it’s architecture seem to allow for an Arch Linux backend without hacks. Hence, there are good reasons to start some competition.
I’m looking forward to Shaman 2!
Well (as I replied to Karl before), this is not an attempt to take over PackageKit, more to complete it. PackageKit is still a first-class supported backend together with Aqpm. Simply here the abstraction level is much higher and GUI-oriented, and creating a backend really makes sense if PackageKit for any reason cannot be supported completely on your packaging system. Otherwise, giving a shot to writing a real packagekit backend would be a better choice.
Really great package manager. The only thing I dislike is that the Package & Package Infomation window are separated. I find this is what KPackageKit handled better. A klick on a package lets expand the entry and shows information about it. Another click lets shrink it again.
The “two-window-solution” is simply to much waste of space for me.
Thanks. We’ll think about that possible approach – it’s definitely an interesting one. Anyway, given the nature of shaman, you could have both of them π I’ll think about this as well
It will be great if all the KDE distributions use same package manager interface.
Thank you for you time and effort. Keep it up.
That’s what we hope π
I’m looking forward to seeing this on my gentoo. Right now i must use a cli based emerge. Not that I complain (emerge is really powerfull) but having a really working manager would be nice π
just to say..I’m trying out shaman2 and it seems cool, it requires aqpm-svn though. Grande Dario, la strada Γ¨ giusta!
Grazie π
This is really good news. KDE is missing a good packagemanagement. About your plugins and backends. I do not know about the technical side. But would it possible to integrate Firefox, Thunderbird and OpenOffice extension management into Shaman? That would be really usefull. I do not like the fact that installation and updating of the system is split up into several apps. It would be great, if this shaman could bring it together again.
It is possible on a theorical level. Shaman implements a special kind of plugins, Providers, which I did not talk about (Lukas will in another post soon). Briefly, Providers allow separate queues of different subsystems with limited possibilities. I don’t know about the internals of these extension, but provider plugins were designed to allow this kind of small actions. So… I never thought about something similar, but what I can say is that the architecture of Shaman allows such a thing to be implemented
I used to think that synaptic was the greatest package management gui ever, until i met shaman. Its only disadvantage is its lacking stability, so hearing of a complete rewrite is like package-mangager-gui-christmas!
I would like to have the package manager integrated into Systemsettings like KPackagekit is, is that planned? A seperate menu entry for package management kind of breaks the KDE structure in my opinion.
Congrats for your awesome work. Cant wait to have it in chakra :))
Thank you a lot for your work, I’ve been strugling to find a good Package Manager until I found Shaman. As whoever said, it only laks stability so a rewrite are really good news!
Keep the great work!
I know, this has already been asked, but does it actually make sense to use the distribution package manager, such as yum or apt, directly, instead of using PackageKit? Given that all the infrastructure for privilege handling is in Shaman, this would possibly strip away one level of abstraction.
/me dreams…… when will a universal package format created? No backends and abstractions then……
[…] Introducing Shaman, a new universal package management frontend Hello people, yes, you can shiver at the title: we have (are still, actually) done a new package management frontend. […] […]
Top Posts « WordPress.com said this on 3 December, 2009 at 2:22 am |
will you try to put Shaman in main KDE branch or ExtraGear or will you remain separate? I would like for this to be integrated in to the main KDE development.
Someone please just make a smart backend. Smart is missing a decent QT based frontend and some argue a decent frontend at all. It would be nice to have options and this looks to be a good option.
hey drfav,
great news. because my current shaman from kdemod crashes when it was supposed to minimize to tray, so I never can use it for updates.
One question/suggestion:
Will it support yaourt/aur integration, because it’s one of the reasons that I don’t use shaman that much for serious package research?
Thansks alot
Woohoo!!! I’ve been waiting forever (or at least since the times of Adept in KDE3) for a good & powerfull package manager for KDE! I can’t stand the new adept and packagekit cause of the lack of features.
Shaman looks really promising! Good luck!
Yaaay this is great !
I would like to ask, are you writing documentation on plugin development ?
And documentation over libshaman, so we can code some other apps using the power of shaman (something like the Software Center of Ubuntu could be better for the end user than something like Synaptic : actually Shaman in the screenshots seems more like a package manager than a software manager… and as I allways wanted to do my own KDE package manager, libshaman could be a painless solution !).
Hey guys!
Yes, Shaman will most likely go to extragear, because we don’t really want to follow the KDE release cycle (and backend devs won’t as well I think…). A decent AUR integration will be up soon I hope…
And yes there will of course be documentation… I’m currently writing the scripting side of things in C++ (expect a blog-post soon)…
Lukas
What about contacting KPackageKit maintainer to join forces? It would be great. Also, we must recognize that KPackageKit loses against GNOME PackageKit (I’m talking about 2 frontends, not about PackageKit).
[…] other project I’m currently busy with is Shaman2. Dario wrote a long blog post about it already, so I won’t say too much. Just one thing: It’s awesome… Expect […]
Major improvements in KGet and continued work on Shaman2 « Boom1992’s Weblog said this on 22 December, 2009 at 4:02 pm |
Can Shaman2 use something similar to Deltarpm so that download size is smaller?
This is very neat!
Hopefully this will be able KDE to have KDevelop IDE with option to make .deb and .rpm4 packages from it. (ideally from a unified interface split in tabs by general and specific options)
Also please make sure the API has notion of downstream upstream package managers, hierarchical package management.
Such an api makes it possible to control the chaos caused by simultaneously running multiple package managers without conflicts. While making it usable because of the unified control the user gets as in one is the upper that controls the others.
The upper one could even divide the tasks and allocate write oppertunities and organize them. When two managers are asking for absolute control, the user should get a notification to make a choice. This notification should be able to come from the upper ones negociating. This would make it possible to have a very neat API in place.
After a year or five you can generalize it and make it ready for Linux Standard Base. Your work could have a tremendously huge positive impact on Linux package managing if one thinks of the big picture, where this API can change things for the better.
Please keep it on and also ask for someone of gnome to do an implementation as well. Thanks.
Also while you’re at it anyway.
Maybe it’s a safeguard to think about update systems.
An api for letting programs update their selves and install/update plug ins from api’s by the system’s package manager stuff.
Would make a lot of things better.
Best Software Downloads and Reviews. the most comprehensive source for free-to-trysoftware downloads on the WebBEST 4 DOWNLOADS