hiteksoftware.com Forum Index hiteksoftware.com
User discussion forum
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

synchronize maximum size

 
Post new topic   Reply to topic    hiteksoftware.com Forum Index -> Frequently Asked Questions
View previous topic :: View next topic  
Author Message
fotoprint



Joined: 16 Oct 2009
Posts: 4

PostPosted: Fri Oct 16, 2009 12:15 pm    Post subject: synchronize maximum size Reply with quote

I'm trying to synchronize a very large amount of data (~2 TB) and the task runs for about 4 minutes, but doesn't transfer any files, exiting with code 0 and no errors in the log. If I reduce the amount of data to several hundred megs it works fine. I'm presuming it's the number of files/folders (several hundred thousand) that's the problem, rather than the actual size.

The suggestions from an old topic were to do it in batches, but it would have to be 20-30 tasks and that's not a practical workaround in this case.
Back to top
View user's profile Send private message
Robert
Site Admin


Joined: 25 May 2006
Posts: 1140

PostPosted: Sat Oct 17, 2009 8:48 pm    Post subject: Reply with quote

Yes.. it is a memory limitation in the synchronize task. The thread runs out of memory for very large number of files/folders.

how many subdirectories are in the main root folder?
you could run a subdirectory loop task. for each subdirectory found, run a synchronization task. this would break up the task into as many batches as there are subdirectories in the main folder.
Back to top
View user's profile Send private message
fotoprint



Joined: 16 Oct 2009
Posts: 4

PostPosted: Mon Oct 19, 2009 6:38 am    Post subject: Reply with quote

Good idea, thank you. I have 26 subdirs in root (one for each letter) and the loop task seems to be doing the job.
Back to top
View user's profile Send private message
fotoprint



Joined: 16 Oct 2009
Posts: 4

PostPosted: Tue Oct 20, 2009 12:09 pm    Post subject: Reply with quote

Ok I *almost* have this working, but having a bit of trouble with target subfolder names.

In the task I have the source folder:

sourcepath/$%loop::CurrentSubdirectoryName%$

which works fine. For the target, I can't seem to get it to use the same structure. I've tried targetpath/$%loop::CurrentSubdirectoryName%$ but it says invalid target path. If I leave the variable off, it just dumps all of the contents into the root target path.

Is there any way I can have the CurrentSubdirectoryName passed to the target path?
Back to top
View user's profile Send private message
Robert
Site Admin


Joined: 25 May 2006
Posts: 1140

PostPosted: Wed Oct 21, 2009 8:40 am    Post subject: Reply with quote

Invalid directory could just mean that the target directory does not exist at the start of the task. As a test , create one of the expected target directories and see if this error goes away, for that directory atleast.

We have changed the code in 8.29 (due monday, oct 26th) to create the target directory if it does not exist. This will be consistent with other tasks where the target directory supports dynamic variables.
Back to top
View user's profile Send private message
bluefishblue



Joined: 04 Nov 2009
Posts: 2

PostPosted: Wed Nov 04, 2009 3:43 pm    Post subject: Can you further explain subdirectory loop task Reply with quote

I think I'm having the same problem. I'm trying to sync my entire home directory to another folder. I tried to set up the sync folders task as a unidirectional mirror. The first time you run it, it works great, but each additional time it runs for 3-5 minutes, has exit code 0, and doesn't copy/update anything.

My home directory has about 10 subdirectories in it, so that subdirectory loop task sounds like it might do the trick. Can anyone further explain how to set it up? Do I have to create a new task for every subdirectory? Is there a way to have it do something like: "look in source directory... if a subdirectory exists, create it in target directory and sync all files" and have it do that for every subdirectory.

thanks
Back to top
View user's profile Send private message
melvin
Site Admin


Joined: 25 May 2006
Posts: 635
Location: Santa Barbara, california

PostPosted: Wed Nov 04, 2009 9:45 pm    Post subject: Reply with quote

Use 8.29 or later.
You would need one Subdirectory loop and one Synchronize task.
lets assume task titles are 'loop' and 'sync'

Run the Subdirectory loop task against the main folder.
Task to run field = sync

In the synchronize task:
Source path = c:\source_folder_path\$loop::CurrentSubdirectoryName%$
Target path = c:\target_folder_path\$loop::CurrentSubdirectoryName%$

then schedule the 'loop' task. for every subdirectory found, it will run the synchronize task and set the source and target directories.
Back to top
View user's profile Send private message Send e-mail
bluefishblue



Joined: 04 Nov 2009
Posts: 2

PostPosted: Tue Nov 10, 2009 11:11 am    Post subject: Reply with quote

Thanks for you help Melvin. I set up my loop and sync tasks as you specified, and they seem to be working great. However, it seems like if any of the subdirectory sync tasks have an exit code other than 0, the entire loop task stops and will not go on to the next subdirectory. Is this by design? Is there any way to fix it so that it will hit all sub directories?

thanks.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    hiteksoftware.com Forum Index -> Frequently Asked Questions All times are GMT - 8 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group