Last week I visited a new customer who is making medical and industrial devices which have one thing in common: image and video capturing, and letting the user save these files onto a USB key. These devices run embedded Linux and the application is done in Qt (and gstreamer for the video capture).
The new requirement, and the reason I was called in, is to be able to save these files over the network, to a samba or a WebDav share. An abstraction framework for copying files over the network to multiple protocols is something I know very well: the KDE framework called KIO implements exactly that, and I happen to be its maintainer. And after the last 4 years of effort in making this much more modular, it is now possible to use KIO with very few dependencies, as we found out.
In this two days on-site visit, I first explained many things about KIO (the API, the framework implementation, the available protocols, etc.), and then we started integrating it into their custom embedded Linux distribution.
The KIO framework consists of 3 libraries (KIOCore, KIOWidgets, KIOFileWidgets) and a number of helper programs (daemons).
However for this use case, only KIOCore is needed. Some user interface is needed:
'error and progress notification, the handling of "file already exists, overwrite/rename/skip?", a dialog for entering user/password for the remote share, a way to display SSL certificate details and approval/refusal...'
But of course for all this, the KDE user interface (implemented by KIOWidgets) isn't desired, rather another implementation that integrates better with the existing embedded GUI. The good thing is that the proper core/gui separation in KIO allows for exactly this: using only KIOCore and implementing some C++ interfaces to provide a customized user interface.
After adding an option to the KIO buildsystem for building only KIOCore, we were able to reduce greatly the number of required dependencies. Only the following are needed: extra-cmake-module (buildsystem):
- KCoreAddons (Job API)
- KConfig (configuration framework)
- KDBusAddons (daemon modules)
- KI18n (translation framework)
- KArchive (for compressed HTTP transfers).
A simple patch commenting out some unneeded protocols for this particular use case allowed us to avoid some additional dependencies (e.g. Solid, for the "trash" implementation).
The dependencies listed above are all "tier 1" frameworks, i.e. they only depend on Qt. So we were able to quickly integrate them into the hand-made Linux distribution, and we have KIOCore ready to use. After copying over the unit tests and setting up a few paths, we had all the tests running without errors.
The final hurdle was the process that handles SSL certificate policies, which was implemented as a module for KDED, a generic process that lives in another framework, bringing in more dependencies. I instead made it standalone, which solved the immediate problem. I'm now working on a better solution - a KIO daemon with multiple modules, for the various service that the KIO framework needs in a separate process (cookie handling, etc.)
On the topic of multiple processes, the customer appreciated greatly the fact that KIO protocol implementations (called "kioslaves") are running in a separate process, because this allows them to look into running them as a different user, to give them limited permissions for security reasons. That user can have a read only access to the saved files (and to nothing else), since the use case is only upload operations.
All this in order to reduce the number of dependencies that KIO has, making it easier than ever to use KIO in any Qt application, so as to benefit from easy to use, asynchronous, well-tested network transparency with more than 90 different protocols available. I'm sure there are other applications with the same need :-)
9 Comments
19 - Feb - 2015
Mike Lothian
Would it be worth looking at splitting the frameworks up more?
The smaller the frameworks, with as few dependencies as possible the more code reuse we would see in other Qt projects
20 - Feb - 2015
markg85
Great post!
I am wondering about the performance though. And i'm not talking about UDSEntry ;)
You surely know that there are quite some bug reports for KIO slaves addressing performance issues in at least smb and ftp. More interesting, it seems to be different per machine. I actually have this same issue. My SMB performance through KIO is... not stellar. The same through a cifs mount is as fast as my network can handle it. That makes me wonder if you found any issues regarding slave performance? Or is it working just fine in the environment where you installed it?
22 - Feb - 2015
David Faure
In these only two days to simplify dependencies and integrate KIO with the existing "distribution", there was no time to test performance. I'm not sure it's a major concern in this case anyway.
22 - Feb - 2015
MihailNaydenov
How does this compare to C++ File System TS ? Is (will) it be able to handle this scenario? Thanks
16 - Sept - 2015
arnold
is there any other lib or framework same as KIO. but more small and light. I just need ssh and ftp.
21 - Dec - 2017
Mostefa
Hi,
Thank you for sharing, i am having the same need (transfer folder from embedded linux to windows shared folder), but sadly i am using Qt4, and i think that KIO work only with Qt5.
Do you know if there is a similar framework working with Qt4 ?
21 - Dec - 2017
David Faure
KIO exists for Qt4 as well, as part of the kdelibs package. This brings in more dependencies though; things have been more clearly separated between Qt4-based kdelibs and Qt5-based KF5. Alternatively you could try using libsmbclient directly, possibly based on the kio_smb code.
2 - Apr - 2020
fevancio
Thank you for sharing! would it can be used in my Qt application that running on windows?
2 - Apr - 2020
David Faure
Yes, KIO works on Windows, and all the unittests pass. Proof, from KDE's continuous integration https://build.kde.org/job/Frameworks/job/kio/job/kf5-qt5%20WindowsMSVCQt5.14/
To use it on Windows, I recommend using Craft in order to fetch, compile and install automatically KIO and all of its dependencies. https://community.kde.org/Craft