Creating remote repo in organization requires use of API. Having thousands of projects will require thousands of repos. That's what I am thinking of the process steps.
Here, I put the unprotected PscEnc.mdb file out in my Google Drive as well. There aren't any macros in it, so it's pretty much just database data.
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
The auto-generated @PSC_ReadMe_Xxx.txt files contain something like this
Code:
Title: [3D maze & frustum ]
Description: 3D "maze" doesn't really have an ending.. but it serves another purpose. This is a cell&portals maze. It uses exact visibility ( recursion ) and clipping to create a view of the maze. I originally had to do this in C++.net using openGL. This USES PURE VB. only a few api calls. I'm looking for someone who has a good idea for a maze building algorithm so i can do some file I/O. right now i just build the maze in the program which takes up A LOT of lines, and could be done much easier by reading a file. anyway, leave your comments, learn, enjoy.
This file came from Planet-Source-Code.com...the home millions of lines of source code
You can view comments on this code/and or vote on it at: http://www.Planet-Source-Code.com/vb/scripts/ShowCode.asp?txtCodeId=57868&lngWId=1
The author may have retained certain copyrights to this code...please observe their request and the law by reviewing all copyright conditions at the above URL.
These will probably be included verbatim in the README.md
For the name of the repo I was thinking about author_name_some_part_of_title so that LaVolpe's submissions would order alphabetically :-)) Could include the submission ID at the end as a number though.
I was thinking about author_name_some_part_of_title so that LaVolpe's submissions would order alphabetically :-))
I was thinking that the author_name_some_part_of_title section should be filtered like, Replace("LaVolpe", "Avid VB Fan") before adding to the repository :-))
Insomnia is just a byproduct of, "It can't be done"
I was thinking that the author_name_some_part_of_title section should be filtered like, Replace("LaVolpe", "Avid VB Fan") before adding to the repository :-))
Don't do it.
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
Found the "secret" and "magic" offset where the bad encryption happens. Even the section is not encrypted (header + some of the data) it could be partially encrypted and that's why my algo failed. Still having 291 files with errors during extraction using SharpZipLib - most are "Cannot find central directory".
The total process to decrypt and extract the data from ZIPs (without writing to temp files - all in memory processing) takes about 15 minutes on my PC using hard drive as storage for ISOs and unpacked data:
Code:
Decrypt and extract projects from CDs
-------------------------------------
Processing CD1 from path e:\
CD2 extraction time taken: 251.371s
Processing CD2 from path f:\
CD3 extraction time taken: 159.1s
Processing CD3 from path g:\
CD4 extraction time taken: 113.134s
Processing CD4 from path h:\
CD5 extraction time taken: 128.607s
Processing CD5 from path i:\
CD6 extraction time taken: 99.211s
Processing CD6 from path j:\
CD7 extraction time taken: 63.033s
Processing CD7 from path k:\
CD8 extraction time taken: 75.378s
Total time taken: 889.839s
Press Enter to exit...
Probably using SSDs and parallelize the processing into multiple threads it may work much faster but it is not important at the moment.
Next step is to process @PSC_Readme* files to get better project names and generate README.md file.
Next step is to process @PSC_Readme* files to get better project names and generate README.md file.
Maybe I should stay out of it, or maybe I'm confused. But isn't that what my ZipFileIndex.tsv and ZipFileIndex.xlsx files are? That's the ZIP file name, title from @PSC_Readme*, and description from @PSC_Readme*. There's not much more in those @PSC_Readme* files.
Anyway, getting it all into a well organized Git repository would be nice, I suppose.
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
Maybe I should stay out of it, or maybe I'm confused. But isn't that what my ZipFileIndex.tsv and ZipFileIndex.xlsx files are? That's the ZIP file name, title from @PSC_Readme*, and description from @PSC_Readme*. There's not much more in those @PSC_Readme* files.
Anyway, getting it all into a well organized Git repository would be nice, I suppose.
Yes, it is what you've done. But to create good looking README.md file requires to add some formatting and insert the image from Data\Pictures directory (still haven't checked the relation of IDs in picture file name).
I can use your files where you extracted all the info (title and description) but this adds another dependency file. My idea is to mount ISOs or copy to separate directories and run the tools to extract data, prepare Git repos and finally to create the repos in centralized Git hosting solution like GitHub.
Some ugly and messy test code for those interested in decryption and extraction process. It will be re-written for the final version. Written in VB.NET (VS2019, targeting .NET Framework 4.8):
VB.NET Code:
Imports System.IO
Imports Extensions.Strings
Imports ICSharpCode.SharpZipLib.Core
Imports ICSharpCode.SharpZipLib.Zip
Public Class PSCAnalyzer
Public Event OnErrorOccurred(s As String)
Private Sub ErrorOccurred(s As String)
RaiseEvent OnErrorOccurred(s)
End Sub
Public Event OnInfoEvent(s As String)
Private Sub InfoEvent(s As String)
RaiseEvent OnInfoEvent(s)
End Sub
Public Event OnLogEvent(s As String)
Private Sub LogEvent(s As String)
RaiseEvent OnLogEvent(s)
End Sub
Private Enum ZIPHeaderType
LocalHeader = 0
CentralDirFileHeader = 1
EndCEntralDirRecord = 2
End Enum
Private ZIP_LOCAL_FILE_HEADER As Byte() = {&H50, &H4B, &H3, &H4}
Private ZIP_CENTRAL_DIR_FILE_HEADER As Byte() = {&H50, &H4B, &H1, &H2}
Private ZIP_END_CENTRAL_DIR_RECORD As Byte() = {&H50, &H4B, &H5, &H6}
Private ENCRYPTED_LOCAL_FILE_HEADER As Byte() = {&HAF, &HB4, &HFC, &HFB}
Private ENCRYPTED_CENTRAL_DIR_FILE_HEADER As Byte() = {&HAF, &HB4, &HFE, &HFD}
Private ENCRYPTED_END_CENTRAL_DIR_RECORD As Byte() = {&HAF, &HB4, &HFA, &HF9}
Private _filename As String
Private _buff As Byte()
Private _buffSize As Integer
Private Sub ExtractZipStream(ByVal rdr As Stream, ByVal destDir As String)
Try
Using zf = New ZipFile(rdr)
For Each zipEntry As ZipEntry In zf
If Not zipEntry.IsFile Then Continue For
Dim zippedFile = zipEntry.Name
Dim fullDestName = Path.Combine(destDir, zippedFile)
Dim outDirName = Path.GetDirectoryName(fullDestName)
If Not outDirName.IsNullOrEmpty Then
Directory.CreateDirectory(outDirName)
End If
Dim buff As Byte() = New Byte(4095) {}
Using zs = zf.GetInputStream(zipEntry)
Using outfs = File.Create(fullDestName)
Try
StreamUtils.Copy(zs, outfs, buff)
Catch ex As Exception
LogEvent($"Error extracting file {zipEntry.Name} from archive {_filename}: {ex.Message}")
There is no time these days but the creation of README.md is done:
If someone knows the relation between picture names (e.g. pic2000101071382022.jpg ) from /data/pictures/ to the project name, please tell so I will not waste time to check those numbers.
If someone knows the relation between picture names (e.g. pic2000101071382022.jpg ) from /data/pictures/ to the project name, please tell so I will not waste time to check those numbers.
All relevant info for the submission is in the PscEnc.mdb incl. picture filename, zip filename, title and description so parsing @PSC_Readme for title/description is a bit useless but nevertheless -- good job!
Extracted from all seven CD codes without zip files, i.e. located in the text format in the database in the "Code" table. They contain many elementary codes, consisting of one or several lines. There are VB, JS, HTML and C ++ codes. Plus - the screenshots attached to them.
Hi to all!
I read on this topic that sadly PSC is gone (i hope not forever)...
I need to download an old shared PSC code from the following link: http://www.planet-source-code.com/vb...68013&lngWId=1
Any chance to find it in one of repositories that some of you kindly share here? And if yes, please anyone can explain me how to find it?
Thanks in advance
EDIT: Nevermind mates, just found it to myself, thanks to Elroy repository (really THANKS to you man for your hard work!!! )
Last edited by RelaAlterEgo; Sep 12th, 2020 at 09:33 AM.
I had few hours and tested Gitea API how to create repository in organization and then initialize new Git repo for each project and push it to server. GitHub API is very similar so I expect the code to work without lots of changes.
What I've found is the push action which takes time and may fail. My tests were with four repos which are bigger than the project from PlanetSourceCode CDs and it took about a minute to create new repo in Gitea (server running on localhost), init local git repo, initial commit of files and the push the origin server.
The whole procedure with thousands of repos will take hours and it requires failures management - connection issues, failed pushes, errors returned by Gitea/GitHub/GitLab API. Blindly retrying failed action is not possible as most of the operations are creating new info: new local repo, organization repo, first (initial) commit, first push.
Most failed operations are easily reversible, e.g. deleting .git dir from local repo, delete remote (Gitea) repo and so on. Probably keeping track of all steps in local SQLite db will be the easiest way to be able to continue the operation even if application crashed or was manually stopped.
Some test VB.NET code to create new org repo using Gitea API, init local Git repo and push it to origin server.
VB.NET Code:
Private Async Function CreateRepoAsync(ByVal orgName As String, ByVal repoName As String) As Task
Using http = New HttpClient()
Dim url = $"http://localhost:3000/api/v1/orgs/{orgName}/repos?token={GITEA_TOKEN}"
Dim reqObj = New CreateOrgRepoRequest() With {
.Name = repoName,
.Description = "Planet-Source-Code repo",
.AutoInit = False,
.DefaultBranch = "master",
.PrivateRepo = False
}
Dim json = JsonConvert.SerializeObject(reqObj)
Dim content As HttpContent = New StringContent(json, Encoding.UTF8, "application/json")
Dim resp = Await http.PostAsync(url, content)
If resp.StatusCode <> HttpStatusCode.Created Then
'
' Handle errors (repo already exists, org not found, etc.)
'
End If
End Using
End Function
Private Sub ExecGitCommand(ByVal workDir As String, ByVal params As String)
Dim startInfo = New ProcessStartInfo("git", params)
I've read the news on HN about the new CLI, but even it can be used for single repo creating, the whole process of errors handling will become more complex when there are thousands of repos. Anyway, I have the routines which manage almost the whole process of transferring PSC CDs to centralized Git repos server so it can be tested and then problems will be reported.
Still 8798 submissions remaining so full upload will take a couple of days at least, during which you can monitor progress on the org home page as repos are being created.
Notice that on github it is possible to limit the full-text search to a particular organization only and that repo names include the original PSC submission Id as embedded in the original PSC link (i.e. ShowCode.asp?txtCodeId={submission_Id_is_here})
Completed a selection from all databases of textual information. In order not to use the bases themselves in the future. The information is stored in several fields and two tables. Getting it without a special program is not convenient. Example:
To get the code for "Title" "!!!! *** Cool Roll *** !!!!" it is necessary to determine id and Worldld in the "Submission" table, then by their numbers in the "Code" table, in the "line" field, find the actual code. My utility collects the values of several fields in two tables and it turns out like this:
Code:
Description :
This is a really cool effect that you can add to make it so that when you click a minimize button, the form slides off the screen and a little mini form appears in the bottom left hand corner. When you click this mini forms title bar, the form appears again. A Very cool effect that I just figured out. (o:
'.................................................................................
More Notes :
Inputs: Follow these instructions.
1) Start up a new project in your VB.
2) Add a new form. (Form2)
3) Add a timer to Form 1 and set it's interval to 1 by pressing F4 to access the properties.
4) Make Form 1's Border Style "0 - None" and make Form 2's Border Style "3 - Fixed Dialog".
5) Add a Label to Form 1. (Label1) Make it's caption "_" (For Minimize)
6) Make Form2's Property "ControlBox = False".
7) Make Form2's Property "Movable = False".
8) And MOST IMPORTANTLY, make Form2's Property "WindowState = "Minimized"
9) That's all, but remember to MINIMIZE YOUR VB WHEN RUNNING THIS CODE. Thanks. (o:
Assumes: none
Code Returns: none
Side Effects: none
Api Declarations: none
'.................................................................................
Code :
'Put this in Form1's General Declarations.
Private Sub Label1_Click()
Timer1.Enabled = True
Form2.Visible = True
End Sub
Private Sub Form_Load()
Timer1.Enabled = False
End Sub
Private Sub Timer1_Timer()
Form1.Top = Form1.Top + 60 'You can adjust the 60 to whatever you prefer. Highering it will make the form drop faster. (o:
Form2.Enabled = True
End Sub
'Put this in Form2's General Declarations. (o:
Private Sub Form_Activate()
Form1.Show
Form2.Hide
Form1.Top = (Screen.Height - Height) / 2
Form1.Left = (Screen.Width - Width) / 2
Form1.Timer1.Enabled = False
End Sub
As a result of the utility operation, 3869 .txt files were obtained.
Due to the fact that many names had invalid characters, they were replaced with "_". Shortest file:
Code:
Description :
Only one line of code, and it works!
'.................................................................................
More Notes :
Inputs: None
Assumes: None
Code Returns: None
Side Effects: None
Api Declarations: None
'.................................................................................
Code :
app.taskvisible = false
Btw, do you keep the titles and the author names of the submissions in these text files?
Edit: Btw, this last submission is a bit brain-dead as the author posted the code in *API Declarations* section instead of the code section on PSC submit page :-))
I only keep the names from the Title field with invalid Windows filenames substitution.
I just forgot about saving the names of the authors.
The challenge was to save the codes.
There are also completely simple codes. I would say advice. For example "Find the current weeknumber easy":
Code:
Description :
This finds the current weeknumber easy.
'.................................................................................
More Notes :
Inputs: None
Assumes: None
Code Returns: None
Side Effects: None
Api Declarations: None
'.................................................................................
Code :
MsgBox DatePart("ww", Date)
There are also quite sensible codes in the CD databases "TV Lines Image Filter":
Code:
Description :
This code puts lines over a picture box's picture. You can set it's opacity, and it's direction.
'.................................................................................
More Notes :
Inputs: PictBox, the picture box to manipulate. optional Opacity. This controls how transparent/solid the lines are. The value works best with a value 1-100. And direction...wich sets the lines horizontal(1) or vertical(2).
Assumes: For this to work, the picturebox must have AutoRedraw set, and it's scalemode must be pixels.
Code Returns: None
Side Effects: if it's a very big image, it might be a tiny bit slow.
Api Declarations: Public Declare Function SetPixel Lib "GDI32" (ByVal HDC As Long, ByVal X As Long, ByVal Y As Long, ByVal crColor As Long) As Long
Public Declare Function GetPixel Lib "GDI32" (ByVal HDC As Long, ByVal X As Long, ByVal Y As Long) As Long
'.................................................................................
Code :
Public Sub TVLines(PictBox As PictureBox, Optional Direction As Integer, Optional Opacity As Long)
Dim i As Long, k As Long, r As Long, g As Long, b As Long, pixel As Long, pix As Long
If IsMissing(Opacity) Then Opacity = 25
If IsMissing(Direction) Then Direction = 1
Opacity = Opacity * 2.55
Opacity = Round(Opacity)
For k = 0 To PictBox.ScaleHeight - 1
For i = 0 To PictBox.ScaleWidth - 1
'get current pixel
pixel = GetPixel(PictBox.HDC, i, k)
'get rgb values of the pixel
r = TakeRGB(pixel, 0)
g = TakeRGB(pixel, 1)
b = TakeRGB(pixel, 2)
'the code alternates lightness/darkness each line
If Direction = 1 Then
pix = k
Else
pix = i
End If
If pix / 2 = Int(pix / 2) Then
r = IIf(r - Opacity < 0, 0, r - Opacity)
g = IIf(g - Opacity < 0, 0, g - Opacity)
b = IIf(b - Opacity < 0, 0, b - Opacity)
Else
r = IIf(r + Opacity > 255, 255, r + Opacity)
g = IIf(g + Opacity > 255, 255, g + Opacity)
b = IIf(b + Opacity > 255, 255, b + Opacity)
End If
'set new pixel
SetPixel PictBox.HDC, i, k, RGB(r, g, b)
Next i
PictBox.Refresh
Next k
PictBox.Refresh
End Sub
'just a function to get rgb values of a pixel
'I borrowed it from Jongmin Baek's Drawer (an exellect program, btw)
Function TakeRGB(Colors As Long, Index As Long) As Long
IndexColor = Colors
Red = IndexColor - Int(IndexColor / 256) * 256: IndexColor = (IndexColor - Red) / 256
Green = IndexColor - Int(IndexColor / 256) * 256: IndexColor = (IndexColor - Green) / 256
Blue = IndexColor
If Index = 0 Then TakeRGB = Red
If Index = 1 Then TakeRGB = Green
If Index = 2 Then TakeRGB = Blue
End Function
Last edited by Argus19; Sep 23rd, 2020 at 12:25 PM.
Apparently, the authors write for beginners so that there are no questions.
It was upsetting that there are 22,598 blocks of text information on 7 CDs, of which only 3869 are unique.
"Trade in expired goods".
So far there are 4429 zip archives missing, meaning there is an entry for the submission in the mdb but the particular zip archive with the sources is missing and in this case the whole repo does not get uploaded to github.
This was to be expected as there are 13k zips only for a total of 22k submissions registered in the mdb.
I'm currently using Elroy's GDrive AllTheZipFilesZipped (before deleting binaries).zip upload but he used your decryptor to "massage" those .psc files into plain zip archives.
Thank. I would like to get .zips after 2014 to combine in one or two DVDs.
Unfortunately, sites sometimes "disappear".
I placed test information extracted from .mdb on Google disk. The existing screenshots have been renamed according to the names of the text files: https://drive.google.com/file/d/1FhZ...ew?usp=sharing
Last edited by Argus19; Sep 23rd, 2020 at 01:18 PM.
JFYI, upload to github's Planet-Source-Code organization is now complete with a total of 16655 repos created and currently all the source code from all the PSC is full-text searchable (including by the original PSC ID).
Besides browsing the source code every repo can be downloaded as a single zip directly from github. Just search for Download ZIP link on repo's front page, git client not required!
Additionally every submission can be commented in repo's issues tab (similar to PSC comments) and/or forked with corrections optionally sent back as Pull Requests. Even if no PRs are sent back the forks being made and additional work being commited is visible in the repos Network graph in Insights tab.
In the next couple of days I'll be updating most of the READMEs to include Category, Level, User Rating and Compatibility as these were included in the database found on PSC CDs.