Kaydet (Commit) ea65b849 authored tarafından Suleyman Poyraz's avatar Suleyman Poyraz

Son duzenlemeler detayli olarak changelogda anlatimistir

üst 4ef7ce70
2018-01-28 Suleyman Poyraz <nipalensisaquila@gmail.com>
* distupdate komutu eklendi:
Upgrade komutu buyuk capta guncellemeler için biraz
tehlikeli kaçıyordu ve yetersiz sayılırdı. Bu yüzden bu
modul eklendi.
* Downgrade silindi:
Downgrade çok vasat duruyor. Upgradeden çarptım ama bence yetersiz
o yüzden yeniden yazılana kadar rafa kaldırdım. çok yakında
yeniden yazıp koyacam. öteki türlü hep aklımdan cıkıyor :)
2018-01-27 Suleyman Poyraz <nipalensisaquila@gmail.com>
* Darara (new tag= inary-1.0_rc1):
-> work build, fetch, update, install functions.
|_ Fixed itembyrepodb.
| Not used true tag. I use "Packages" but real tag was "Package".
|_ Fixed fetcher
| now work very well.
|_ Fixed all po files
| Usable Languages are: German, English, Franchis, Turkish and Russian
|_ Clean non useful imports
|_ Fixed filesdb
\ we use leveldb based filesdb (marcin)
2018-01-26 Suleyman Poyraz <nipalensisaquila@gmail.com>
* Python3 kullanımı ile ilgili çok vasat bir kodlama
anlayışımız var:
a) Bu sorun ile ilgili olarak
"%s" ile taglanma yerine "{}" kullaniliyor.
b) Ceşitli düzenlemeler yapıldı. Buna uygun olarak
po dosyaları ve actions.py betikleri (repodan)
düzenlendi.
c) yanlış kullanımlar gereksiz importlar silindi
2018-01-20 Suleyman Poyraz <nipalensisaquila@gmail.com>
* Testler elden geçirilmeli:
-> Testler düzenlendi (2018-01-20)
-> Testler gruplandirildi (2018-01-22)
-> oo ve autoxml testleri eklendi (2018-01-22)
-> testlerin islevselliginden emin olunuldu (2018-01-23)
2018-01-17 Suleyman Poyraz <nipalensisaquila@gmail.com>
* Added system_literals and now work crossplatorm
inary packaging:
Look more about inary/system_literals/Modules.md
2018-01-13 Suleyman Poyraz <nipalensisaquila@gmail.com>
* Scom dosyalari pakette yok!!:
Bu hata Fixe edilmiştir
......
......@@ -22,7 +22,6 @@ Legend:
+ Ruby Language Tools
+ Perl Language Tools
+ CMake Tools
/ AfterInstall Modules
/ unit test
+ source building
......@@ -35,7 +34,7 @@ Legend:
+ command line interface
+ inary-build
+ inary-install
+ inary-index (but not indexing build repos-- Look for errors FIXME)
+ inary-index
+ inary-updatedb
+ single repository index
+ simple dependency checking
......@@ -66,11 +65,10 @@ Legend:
+ implement source database (eray)
+ easy package preparation
? convert ebuild to inary
/ Configuring system interface after installing
+ Configuring system interface after installing
+ what do we need to specify in a package
+ package install: register script and configure system with it
? package remove: unregister script and reconfigure systen with it
(I don't know How I can remove unused config files, databases and others...)
+ package remove: unregister script and reconfigure systen with it (Zaryoh)
+ internet installation
+ support URI's whereever a filename is supported
+ http server
......@@ -91,10 +89,12 @@ Legend:
+ list type
+ class type
? localtext/text types (localtext.... Ummm?.... )
+ write meaningful localtext datas (Zaryob)
- return meaningful localtext summaries and descriptions (Hak getire)
+ revised interface
/ Error handling
+ improved error handling, check routines, hooks, format/print routines
/ write a meaningful traceback output (leş gibi yaHu şu hali ils)
/ write a meaningful traceback output
+ xmlfile support (read/write)
+ string member, in addition to tag/attribute members
+ replace specfile with new specfile using autoxml
......@@ -122,14 +122,14 @@ Legend:
+ ciksemel rewrite and adding python3 support (aquila)
/ partial caching and automatic resume for file download (meren)
+ use a '.part' extension
- resume file downloading (Darara.... fetcher will rewrite with urllib.requests)
- resume file downloading (Darara.... fetcher will rewrite with httplib) (Zaryob)
+ search-file command (eray)
+ exception handling (eray)
+ uniform exception hierarchy
+ proper handling
+ API: we even have application users (eray)
+ a facade module inary.api (eray)
- clean unuseful inary.api functions (example: buildfarm functions move to buildfarm)
+ clean unuseful inary.api functions
+ a library init command (eray)
+ high-level package ops, index (eray)
+ info/util commands (eray)
......@@ -137,9 +137,12 @@ Legend:
+ documentation
+ actionsAPI documentation, unittests (caglar - meren)
+ versioning information document (eray - baris)
+ UI
- INARYMAT
- Automatic dependency finder
/ Automatic dependency finder
/ read pkgconfig files and find dependencies (Zaryob)
+ find cycle dependencies (Zaryob)
- Automatic build upgrader
- pykde & pygtk GUI (cartman)
+ improve interface (eray)
......@@ -156,6 +159,8 @@ Legend:
/ translate messages mesajların çevirisinin yapılması (inary tr.po) (eray)
+ proper unicode handling in code - tons of fixes (eray - baris)
/ configure-pending (eray) (should return which packages configured)
- report scom scripts problems to developers
+ configure pendings (Zaryob)
+ proper commar connection (baris - gurer)
+ ignore-deps (eray)
+ more robust upgrade (eray)
......@@ -222,6 +227,9 @@ Legend:
- incremental build/fetch for repository index (pisi-index.xml)
/ xdelta support (caglar)
- diffsets
/ a useful distupdate command
+ distupdate planner
- distupdate maker
- a command (check-repo) to check if repos are consistent wrt dep relations
- component improvements (optional)
- maintain a list of components each component has
......
* Write a new conf manager:
I write in cosmetical part with curses. In tomorrow I will add maker part
......@@ -39,8 +39,8 @@ import inary.file
from inary.reactor import *
import inary.operations.build
import inary.operations.check
import inary.operations.downgrade
import inary.operations.emerge
import inary.operations.distupdate
import inary.operations.install
import inary.operations.history
import inary.operations.helper
......
# -*- coding:utf-8 -*-
#
# Copyright (C) 2016 - 2017, Suleyman POYRAZ (Zaryob)
#
# This program is free software; you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free
# Software Foundation; either version 2 of the License, or (at your option)
# any later version.
#
# Please read the COPYING file.
#
import optparse
import gettext
__trans = gettext.translation('inary', fallback=True)
_ = __trans.gettext
import inary.cli.command as command
import inary.context as ctx
import inary.reactor as Reactor
import inary.db
class DistUpdate(command.PackageOp, metaclass=command.autocommand):
__doc__ = _("""Update the system a new release
Usage: DistUpdate [ next_dist_release_repo_url ] [ dist-update-list ]
WARNING: DIST-UPDATE risk içerir.
Dist-Update yapmadan önce iki kez düşününüz. Çünkü
Dist-Update sonrası sisteminiz çalışmayabilir, tüm
dosyalarınızı kaybedebilirsiniz.
Sisteminizi Dist-Update yapmadan önce yedekleyiniz ve
EĞER YAPMAK İSTEDİĞİNİZE GERÇEKTEN EMİNSENİZ BU İŞLEMİ
BAŞLATINIZ.
OLUŞAN HİÇBİR DİSK VE SİSTEM
HASARINDAN SULIN TOPLULUĞU
MESUL DEĞİLDİR
""")
def __init__(self, args):
super(DistUpdate, self).__init__(args)
name = ("dist-update", "distup")
def options(self):
group = optparse.OptionGroup(self.parser, _("dist-update options"))
super(Upgrade, self).options(group)
group.add_option("--security-only", action="store_true",
default=False, help=_("Security related package upgrades only"))
group.add_option("-x", "--exclude", action="append",
default=None, help=_("When upgrading system, ignore packages and components whose basenames match pattern."))
group.add_option("--exclude-from", action="store",
default=None,
help=_("When upgrading system, ignore packages "
"and components whose basenames match "
"any pattern contained in file."))
group.add_option("-s", "--compare-sha1sum", action="store_true",
default=False, help=_("compare sha1sum repo and installed packages"))
self.parser.add_option_group(group)
def run(self):
self.init()
argv = []
argv.extend(self.args)
targetrepo = argv[1]
Reactor.distupdate(targetrepo)
# -*- coding:utf-8 -*-
#
# Copyright (C) 2016 - 2017, Suleyman POYRAZ (Zaryob)
#
# This program is free software; you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free
# Software Foundation; either version 2 of the License, or (at your option)
# any later version.
#
# Please read the COPYING file.
#
import optparse
import gettext
__trans = gettext.translation('inary', fallback=True)
_ = __trans.gettext
import inary.cli.command as command
import inary.context as ctx
import inary.reactor as Reactor
import inary.db
class Downgrade(command.PackageOp, metaclass=command.autocommand):
__doc__ = _("""Downgrade INARY packages
Usage: Downgrade [<package1> <package2> ... <packagen>]
<packagei>: package name
Downgrades the entire system if no package names are given
You may use only package names to specify packages because
the package downgrade operation is defined only with respect
to repositories. If you have specified a package name, it
should exist in the package repositories. If you just want to
reinstall a package from a INARY file, use the install command.
You can also specify components instead of package names, which will be
expanded to package names.
""")
def __init__(self, args):
super(Downgrade, self).__init__(args)
name = ("downgrade", "dg")
def options(self):
group = optparse.OptionGroup(self.parser, _("downgrade options"))
super(Downgrade, self).options(group)
group.add_option("--security-only", action="store_true",
default=False, help=_("Security related package downgrades only"))
group.add_option("-b", "--bypass-update-repo", action="store_true",
default=False, help=_("Do not update repositories"))
group.add_option("--ignore-file-conflicts", action="store_true",
default=False, help=_("Ignore file conflicts"))
group.add_option("--ignore-package-conflicts", action="store_true",
default=False, help=_("Ignore package conflicts"))
group.add_option("-c", "--component", action="append",
default=None, help=_("Downgrade component's and recursive components' packages"))
group.add_option("-r", "--repository", action="store",
type="string", default=None, help=_('Name of the to be downgraded packages\' repository'))
group.add_option("-f", "--fetch-only", action="store_true",
default=False, help=_("Fetch downgrades but do not install."))
group.add_option("-x", "--exclude", action="append",
default=None, help=_("When downgrading system, ignore packages and components whose basenames match pattern."))
group.add_option("--exclude-from", action="store",
default=None,
help=_("When downgrading system, ignore packages "
"and components whose basenames match "
"any pattern contained in file."))
group.add_option("-s", "--compare-sha1sum", action="store_true",
default=False, help=_("compare sha1sum repo and installed packages"))
self.parser.add_option_group(group)
def run(self):
if self.options.fetch_only:
self.init(database=True, write=False)
else:
self.init()
if not ctx.get_option('bypass_update_repo'):
ctx.ui.info(_('Updating repositories'))
repos = Reactor.list_repos()
Reactor.update_repos(repos)
else:
ctx.ui.info(_('Will not update repositories'))
repository = ctx.get_option('repository')
components = ctx.get_option('component')
packages = []
if components:
componentdb = inary.db.componentdb.ComponentDB()
for name in components:
if componentdb.has_component(name):
if repository:
packages.extend(componentdb.get_packages(name, walk=True, repo=repository))
else:
packages.extend(componentdb.get_union_packages(name, walk=True))
packages.extend(self.args)
Reactor.downgrade(packages, repository)
......@@ -28,7 +28,7 @@ import inary.cli.clean
import inary.cli.configurepending
import inary.cli.deletecache
import inary.cli.delta
import inary.cli.downgrade
import inary.cli.distupdate
import inary.cli.emerge
import inary.cli.emergeup
import inary.cli.fetch
......
#!/usr/bin/python
# -*- coding: utf-8 -*-
#
# Dist Update planner using inary api and ciksemel
# This script resides because we need to check if we can
# update, to find out what is missing for update etc.
# without adding new distro repo / upgrading any packages
#
import os
import sys
import bz2
import lzma
import urllib.request, urllib.error, urllib.parse
import ciksemel
import inary
import inary.context as ctx
defaultForceInstallPackageURI = "http://packages.sulin.org/main/force-install.list"
class DistupdatePlanner:
def __init__(self, nextRepoUri, forceInstallUri=defaultForceInstallPackageURI, Debug=False):
self.debug = Debug
self.forceInstallUri = forceInstallUri
self.nextRepoUri = nextRepoUri
self.nextRepoRaw = None
self.nextRepoPackages = {}
self.nextRepoReplaces = {}
self.nextRepoObsoletes = []
self.installedPackages = []
self.packagesToInstall = []
self.forceInstallPackages = []
self.missingPackages = []
self.graphobj = None
self.sizeOfInstalledPackages = 0
self.sizeOfInstalledPackagesAfterUpdate = 0
self.sizeOfPackagesToDownload = 0
self.sizeOfBiggestPackage = 0
self.sizeOfNeededTotalSpace = 0
def uniq(self, i):
return list(set(i))
def getIndex(self):
uri = self.nextRepoUri
ctx.ui.info(_("* Getting index from {}").format(uri))
try:
if "://" in uri:
rawdata = urllib.request.urlopen(uri).read()
else:
rawdata = open(uri, "r").read()
except IOError:
ctx.ui.info("could not fetch {}".format(uri))
return None
if uri.endswith("bz2"):
data = bz2.decompress(rawdata)
elif uri.endswith("xz") or uri.endswith("lzma"):
data = lzma.decompress(rawdata)
else:
data = rawdata
ctx.ui.info(_(" done"))
self.nextRepoRaw = data
def convertToInaryRepoDB(self, ix):
ctx.ui.info(_("* Converting package objects to db object"),noln=True)
doc = ciksemel.parseString(ix)
dbobj = inary.index.Index()
dbobj.decode(doc, [])
ctx.ui.info(" done")
return dbobj
def parseRepoIndex(self):
ctx.ui.info(_("* Parsing package properties in new repo"))
pkgprop = {}
obsoletelist = []
ix = ciksemel.parseString(self.nextRepoRaw)
obsoleteParent = ix.getTag("Distribution").getTag("Obsoletes")
for node in obsoleteParent.tags("Package"):
obsoletelist.append(node.firstChild().data())
for i in ix.tags("Package"):
replaceslist = []
pkgName = i.getTagData("Name")
pkgURI = i.getTagData("PackageURI")
pkgSize = int(i.getTagData("PackageSize"))
pkgHash = i.getTagData("PackageHash")
pkgInstalledSize = int(i.getTagData("InstalledSize"))
replacedPackages = i.getTag("Replaces")
if replacedPackages:
for replaced in replacedPackages.tags("Package"):
replaceslist.append(replaced.firstChild().data())
pkgprop[pkgName] = [replaceslist, pkgURI, pkgSize, pkgInstalledSize, pkgHash]
ctx.ui.info(_(" found {0} packages and {1} obsoletelist").format(len(list(pkgprop.keys())), len(obsoletelist)))
self.nextRepoPackages = pkgprop
self.nextRepoObsoletes = obsoletelist
def getInstalledPackages(self):
ctx.ui.info(_("* Getting installed packages"))
a = inary.reactor.list_installed()
a.sort()
ctx.ui.info(_(" found {} packages").format(len(a)))
self.installedPackages = a
def getForceInstallPackages(self):
ctx.ui.info(_("* Getting force install packages"))
pkglist = urllib.request.urlopen(self.forceInstallUri).read().split()
ctx.ui.info(_(" found {} packages").format(len(pkglist)))
self.forceInstallPackages = pkglist
def calculateInstalledSize(self):
ctx.ui.info(_("* Calculating disk space installed packages are using"))
totalsize = 0
idb = inary.db.installdb.InstallDB()
for i in self.installedPackages:
pkg = idb.get_package(i)
totalsize += pkg.installedSize
#print "%-30s %s" % (pkg.name, pkg.installedSize)
ctx.ui.info(_(" total size = {}").format(totalsize))
self.sizeOfInstalledPackages = totalsize
def calculateNextRepoSize(self):
ctx.ui.info(_("* Calculating package size and installed size of new packages"))
for p in self.packagesToInstall:
i = self.nextRepoPackages[p]
self.sizeOfPackagesToDownload += i[2]
self.sizeOfInstalledPackagesAfterUpdate += i[3]
if i[2] > self.sizeOfBiggestPackage:
self.sizeOfBiggestPackage = i[2]
ctx.ui.info(_(" total download size = {}").format(self.sizeOfPackagesToDownload))
ctx.ui.info(_(" total install size = {}").format(self.sizeOfInstalledPackagesAfterUpdate))
def calculeNeededSpace(self):
ctx.ui.info(_("* Calculating needed space for distupate"))
neededspace = 0
neededspace += self.sizeOfPackagesToDownload
neededspace += (self.sizeOfInstalledPackagesAfterUpdate - self.sizeOfInstalledPackages)
neededspace += 2 * self.sizeOfBiggestPackage
self.sizeOfNeededTotalSpace = neededspace
ctx.ui.info(_(" total needed space = {}").format(neededspace))
def findMissingPackages(self):
ctx.ui.info(_("* Calculating package differences of old and new repos"))
pkglist = []
replacedBy = {}
neededPackages = []
# we make a cache of replaced packages, not to iterate over and over on package dict
for i in self.nextRepoPackages:
pkglist.append(i)
for r in self.nextRepoPackages[i][0]:
pkglist.append(r)
if r in replacedBy:
replacedBy[r].append(i)
else:
replacedBy[r] = [i]
self.nextRepoReplaces = replacedBy
# and we package list of removed (replaced + obsoleted) packages
pkglist.extend(self.nextRepoObsoletes)
uniqpkglist = self.uniq(pkglist)
for i in self.installedPackages:
if i not in uniqpkglist:
neededPackages.append(i)
ctx.ui.info(_(" found {} obsoleted and replaced packages").format(len(neededPackages)))
self.missingPackages = neededPackages
def resolveDependencies(self, A, pkgdb):
ctx.ui.info(_("* Find dependencies for packages to be installed"))
# this would be the system package db on a normal scenario
# packagedb = inary.db.packagedb.PackageDB()
repodict = dict((pkg.name, pkg) for pkg in pkgdb.packages)
# our lovely fake package db, we need to make up one since
# we are working on a repository that is not added to system
class PackageDB:
def get_package(self, key, repo = None):
return repodict[str(key)]
packagedb = PackageDB()
# write package names as a list for testing
# A = repodict.keys()
# construct G_f
G_f = inary.pgraph.PGraph(packagedb)
# find the install closure graph of G_f by package
# set A using packagedb
for x in A:
G_f.add_package(x)
B = A
while len(B) > 0:
Bp = set()
for x in B:
pkg = packagedb.get_package(x)
for dep in pkg.runtimeDependencies():
if not dep.package in G_f.vertices():
Bp.add(str(dep.package))
G_f.add_dep(x, dep)
B = Bp
installOrder = G_f.topological_sort()
installOrder.reverse()
return G_f, installOrder
def planDistUpdate(self):
ctx.ui.info(_("* Planning the whole distupdate process"))
# installed packages
currentPackages = self.installedPackages
toRemove = []
toAdd = []
# handle replaced packages
for i in currentPackages:
if i in self.nextRepoReplaces:
toRemove.append(i)
toAdd.extend(self.nextRepoReplaces[i])
# remove "just obsoleted" packages
toRemove.extend(self.nextRepoObsoletes)
# this should never happen with normal usage, yet we need to calculate the scenario
toRemove.extend(self.missingPackages)
# new packages needed for the new distro version
self.getForceInstallPackages()
toAdd.extend(self.forceInstallPackages)
currentPackages.extend(toAdd)
currentPackages = list(set(currentPackages) - set(toRemove))
indexstring = self.nextRepoRaw
repodb = self.convertToInaryRepoDB(indexstring)
self.graphobj, self.packagesToInstall = self.resolveDependencies(currentPackages, repodb)
ctx.ui.info(" found {} packages to install".format(len(self.packagesToInstall)))
def plan(self):
ctx.ui.info(_("* Find missing packages for distupdate "))
self.getIndex()
self.parseRepoIndex()
self.getInstalledPackages()
self.calculateInstalledSize()
self.findMissingPackages()
self.planDistUpdate()
self.calculateNextRepoSize()
self.calculeNeededSpace()
class MakeDistUpdate():
pass
This diff is collapsed.
......@@ -331,16 +331,6 @@ def fetch(packages=[], path=os.path.curdir):
fetcher.fetch_url(url, path, ctx.ui.Progress)
@locked
def downgrade(packages=[], repo=None):
"""
Downgrades the given packages, if no package given downgrades all the packages
@param packages: list of package names -> list_of_strings
@param repo: name of the repository that only the packages from that repo going to be downgraded
"""
inary.db.historydb.HistoryDB().create_history("downgrade")
return inary.operations.downgrade.downgrade(packages, repo)
@locked
def upgrade(packages=[], repo=None):
"""
......@@ -351,6 +341,28 @@ def upgrade(packages=[], repo=None):
inary.db.historydb.HistoryDB().create_history("upgrade")
return inary.operations.upgrade.upgrade(packages, repo)
def distupdate(targetrepo):
weddingplanner = inary.operations.distupdate.DistupdatePlanner(nextRepoUri=targetrepo, Debug=True)
weddingplanner.plan()
ctx.ui.info(inary.util.colorize(_("*** Conclusion ***"),"red"))
if len(weddingplanner.missingPackages):
ctx.ui.info(_(" found packages preventing distupdate"))
ctx.ui.info(weddingplanner.missingPackages)
else:
ctx.ui.info(_(" distupdate is good to go"))
ctx.ui.info(_(" installed size {}").format(weddingplanner.sizeOfInstalledPackages))
ctx.ui.info(_(" installed size after update {}").format(weddingplanner.sizeOfInstalledPackagesAfterUpdate))
ctx.ui.info(_(" download size {}").format(weddingplanner.sizeOfPackagesToDownload))
ctx.ui.info(_(" biggest package size {}").format(weddingplanner.sizeOfBiggestPackage))
ctx.ui.info(_(" total space needed for distupdate {}").format(weddingplanner.sizeOfNeededTotalSpace))
if ctx.ui.confirm(str(_('Do you want make dist-update?'))):
inary.operations.distupdate.MakeDistUpdate()
@locked
def remove(packages, ignore_dependency=False, ignore_safety=False):
"""
......
......@@ -92,21 +92,20 @@ def get_kernel_option(option):
return args
def get_cpu_count():
"""
This function part of portage
Copyright 2015 Gentoo Foundation
Distributed under the terms of the GNU General Public License v2
Using:
Try to obtain the number of CPUs available.
@return: Number of CPUs or None if unable to obtain.
"""
try:
import multiprocessing
return multiprocessing.cpu_count()
except (ImportError, NotImplementedError):
return None
"""
This function part of portage
Copyright 2015 Gentoo Foundation
Distributed under the terms of the GNU General Public License v2
Using:
Try to obtain the number of CPUs available.
@return: Number of CPUs or None if unable to obtain.
"""
try:
import multiprocessing
return multiprocessing.cpu_count()
except (ImportError, NotImplementedError):
return None
def get_vm_info():
vm_info = {}