- devel - mailman3.cels.anl.gov

Load imbalance in MPIR_Reduce_redscat_gather
by Mikhail Kurnosov 29 Mar '17

29 Mar '17

Hello, In the current implementation of Rabenseifner’s algorithm (MPIR_Reduce_redscat_gather) for MPI_Reduce there is a load imbalance across the processes. The reason is non-uniform distribution of count elements across pof2 processes on the reduce-scatter and the gather steps. The last process pof2-1 is loaded more than others: for (i=0; i<(pof2-1); i++) cnts[i] = count/pof2; cnts[pof2-1] = count - (count/pof2)*(pof2-1); For example, commsize = 13, count=13; pof2=8, cnts[0..6] = 1, cnts[7] = 6. commsize = 1600, count=100000; pof2=1024, cnts[0..1022] = 97, cnts[1023] = 769. In addition, initial calculation of the cnts[i] and disps[i], before reduce-scatter step, takes O(log(p)) time. Summation of the cnts[i] on each step of the reduce-scatter also takes O(log(p)) time. Storing the cnts[i] and displs[i] requires O(p) bytes of memory. I implemented a new version of MPIR_Reduce_redscat_gather following the description in the original papers [1, 2]. Input vectors are halving uniformly, without using of arrays cnts[i] and disps[i]. Thus, the memory consumption and computational complexity is reduced. [1] Rajeev Thakur, Rolf Rabenseifner and William Gropp. Optimization of Collective Communication Operations in MPICH // The Int. Journal of High Performance Computing Applications. Vol 19, Issue 1, pp. 49--66. [2] http://www.hlrs.de/mpi/myreduce.html. With best regards, Mikhail Kurnosov -- Computer Systems Department Siberian State University of Telecommunications and Information Sciences Address: 630102, 86 Kirova str., Novosibirsk, Russia WWW: www.mkurnosov.net

1 0

Re: [mpich-devel] Deadlock when using MPIX_Grequest interface
by Jeff Hammond 09 Mar '17

09 Mar '17

On Wed, Mar 8, 2017 at 5:57 PM, Latham, Robert J. <robl(a)mcs.anl.gov> wrote: > On Wed, 2017-03-08 at 15:17 -0800, Jeff Hammond wrote: > > > > > > On Wed, Mar 8, 2017 at 2:25 PM, Latham, Robert J. <robl(a)mcs.anl.gov> > > wrote: > > > On Wed, 2017-03-08 at 09:25 +0000, Giuseppe Congiu wrote: > > > > Hello Rob, > > > > > > > > > I'm excited to see someone using the MPIX_Grequest interface. > > > We > > > > > used > > > > > the MPIX_Grequest interface to implement non-blocking > > > collective > > > > > I/O, > > > > > and had some bad interactions between libc's aio and the > > > grequest > > > > > callbacks. I don't know if you are running into something > > > similar. > > > > > > > > Maybe. Do you have a description of the problem somewhere? > > > > > > The guy who did that work just left last Friday. I'll have to dig > > > up > > > the archives. Looks like it was a hard-to-debug segfault https://t > > > rac. > > > mpich.org/projects/mpich/ticket/2201 > > > > > > > That link is 404. Not sure if temporary or permanent. > > Huh. Did it word-wrap in a strange way? > > https://trac.mpich.org/projects/mpich/ticket/2201 > > It's back now. I guess Argonne hosting is inconsistent this week. Jeff > ==rob > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] Deadlock when using MPIX_Grequest interface
by Latham, Robert J. 08 Mar '17

08 Mar '17

On Wed, 2017-03-08 at 15:17 -0800, Jeff Hammond wrote: > > > On Wed, Mar 8, 2017 at 2:25 PM, Latham, Robert J. <robl(a)mcs.anl.gov> > wrote: > > On Wed, 2017-03-08 at 09:25 +0000, Giuseppe Congiu wrote: > > > Hello Rob, > > > > > > > I'm excited to see someone using the MPIX_Grequest interface. > > We > > > > used > > > > the MPIX_Grequest interface to implement non-blocking > > collective > > > > I/O, > > > > and had some bad interactions between libc's aio and the > > grequest > > > > callbacks. I don't know if you are running into something > > similar. > > > > > > Maybe. Do you have a description of the problem somewhere? > > > > The guy who did that work just left last Friday. I'll have to dig > > up > > the archives. Looks like it was a hard-to-debug segfault https://t > > rac. > > mpich.org/projects/mpich/ticket/2201 > > > > That link is 404. Not sure if temporary or permanent. Huh. Did it word-wrap in a strange way? https://trac.mpich.org/projects/mpich/ticket/2201 ==rob

1 0

Re: [mpich-devel] Deadlock when using MPIX_Grequest interface
by Raffenetti, Kenneth J. 08 Mar '17

08 Mar '17

Ticket numbers are maintained on GitHub: https://github.com/pmodels/mpich/issues/2201 Ken On Mar 8, 2017 5:17 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: On Wed, Mar 8, 2017 at 2:25 PM, Latham, Robert J. <robl(a)mcs.anl.gov<mailto:[email protected]>> wrote: On Wed, 2017-03-08 at 09:25 +0000, Giuseppe Congiu wrote: > Hello Rob, > > > I'm excited to see someone using the MPIX_Grequest interface. We > > used > > the MPIX_Grequest interface to implement non-blocking collective > > I/O, > > and had some bad interactions between libc's aio and the grequest > > callbacks. I don't know if you are running into something similar. > > Maybe. Do you have a description of the problem somewhere? The guy who did that work just left last Friday. I'll have to dig up the archives. Looks like it was a hard-to-debug segfault https://trac. mpich.org/projects/mpich/ticket/2201<http://mpich.org/projects/mpich/ticket/2201> That link is 404. Not sure if temporary or permanent. Jeff > > > Do you have any desire or plans to submit these changes into > > upstream > > ROMIO? > > The idea would be to push these changes to upstream ROMIO if this is > relevant for the community. I don't encounter many BeeGFS users, but ROMIO file system drivers are fairly self-contained and it wouldn't be a burden to ship with them in ROMIO. > In principle here I have the same intent. The difference is that I > cannot check on progress since > BeeGFS does not provide a way for checking the status of a single > request. Instead it only > offers a blocking wait interface for all the requests submitted for a > certain file (identified > by the filename). Thus I need to invoke deeper_cache_flush_wait() > from inside one of the > callbacks. Blocking the progress engine when it expects to repeatedly call non- blocking functions could work as long as deeper_cache_flush_wait() eventually finishes and nothing needs MPI. Now, all I know about DEEP-ER is what I just read on https://www.beegfs .com/wiki/cacheAPI, so I'm sure I don't know the whole picture, but can you call deeper_cache_flush_is_finished() and deeper_cache_flush() without the WAIT flag? Stick those two routines in the poll_fn(). The generalized request extensions provide a wait_fn() that should be able to handle this, too... When it gets stuck, what does the call stack look like? Is it stuck for good or just making progress really really slowly? ==rob _______________________________________________ To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/devel -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/

1 0

Re: [mpich-devel] Deadlock when using MPIX_Grequest interface
by Jeff Hammond 08 Mar '17

08 Mar '17

On Wed, Mar 8, 2017 at 2:25 PM, Latham, Robert J. <robl(a)mcs.anl.gov> wrote: > On Wed, 2017-03-08 at 09:25 +0000, Giuseppe Congiu wrote: > > Hello Rob, > > > > > I'm excited to see someone using the MPIX_Grequest interface. We > > > used > > > the MPIX_Grequest interface to implement non-blocking collective > > > I/O, > > > and had some bad interactions between libc's aio and the grequest > > > callbacks. I don't know if you are running into something similar. > > > > Maybe. Do you have a description of the problem somewhere? > > The guy who did that work just left last Friday. I'll have to dig up > the archives. Looks like it was a hard-to-debug segfault https://trac. > mpich.org/projects/mpich/ticket/2201 > > That link is 404. Not sure if temporary or permanent. Jeff > > > > > Do you have any desire or plans to submit these changes into > > > upstream > > > ROMIO? > > > > The idea would be to push these changes to upstream ROMIO if this is > > relevant for the community. > > I don't encounter many BeeGFS users, but ROMIO file system drivers are > fairly self-contained and it wouldn't be a burden to ship with them in > ROMIO. > > > > In principle here I have the same intent. The difference is that I > > cannot check on progress since > > BeeGFS does not provide a way for checking the status of a single > > request. Instead it only > > offers a blocking wait interface for all the requests submitted for a > > certain file (identified > > by the filename). Thus I need to invoke deeper_cache_flush_wait() > > from inside one of the > > callbacks. > > Blocking the progress engine when it expects to repeatedly call non- > blocking functions could work as long as deeper_cache_flush_wait() > eventually finishes and nothing needs MPI. > > Now, all I know about DEEP-ER is what I just read on https://www.beegfs > .com/wiki/cacheAPI, so I'm sure I don't know the whole picture, but > can you call deeper_cache_flush_is_finished() and deeper_cache_flush() > without the WAIT flag? Stick those two routines in the poll_fn(). > > The generalized request extensions provide a wait_fn() that should be > able to handle this, too... > > When it gets stuck, what does the call stack look like? > > Is it stuck for good or just making progress really really slowly? > > ==rob > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] Deadlock when using MPIX_Grequest interface
by Latham, Robert J. 08 Mar '17

08 Mar '17

On Wed, 2017-03-08 at 09:25 +0000, Giuseppe Congiu wrote: > Hello Rob, > > > I'm excited to see someone using the MPIX_Grequest interface. We > > used > > the MPIX_Grequest interface to implement non-blocking collective > > I/O, > > and had some bad interactions between libc's aio and the grequest > > callbacks. I don't know if you are running into something similar. > > Maybe. Do you have a description of the problem somewhere? The guy who did that work just left last Friday. I'll have to dig up the archives. Looks like it was a hard-to-debug segfault https://trac. mpich.org/projects/mpich/ticket/2201 > > > Do you have any desire or plans to submit these changes into > > upstream > > ROMIO? > > The idea would be to push these changes to upstream ROMIO if this is > relevant for the community. I don't encounter many BeeGFS users, but ROMIO file system drivers are fairly self-contained and it wouldn't be a burden to ship with them in ROMIO. > In principle here I have the same intent. The difference is that I > cannot check on progress since > BeeGFS does not provide a way for checking the status of a single > request. Instead it only > offers a blocking wait interface for all the requests submitted for a > certain file (identified > by the filename). Thus I need to invoke deeper_cache_flush_wait() > from inside one of the > callbacks. Blocking the progress engine when it expects to repeatedly call non- blocking functions could work as long as deeper_cache_flush_wait() eventually finishes and nothing needs MPI. Now, all I know about DEEP-ER is what I just read on https://www.beegfs .com/wiki/cacheAPI, so I'm sure I don't know the whole picture, but can you call deeper_cache_flush_is_finished() and deeper_cache_flush() without the WAIT flag? Stick those two routines in the poll_fn(). The generalized request extensions provide a wait_fn() that should be able to handle this, too... When it gets stuck, what does the call stack look like? Is it stuck for good or just making progress really really slowly? ==rob

1 0

Re: [mpich-devel] mpich.org is downforeveryone[andnotjustme].com
by Kenneth Raffenetti 08 Mar '17

08 Mar '17

Jeff's mail was delayed over an hour in the ANL network. I suspect that was also the cause of the outage he saw, but is now resolved :). Ken On 03/08/2017 08:57 AM, Halim Amer wrote: > I get a different response than you, Jeff. The site says mpich.org is up. > > Halim > www.mcs.anl.gov/~aamer > > On 3/8/17 7:37 AM, Jeff Hammond wrote: >> http://www.downforeveryoneorjustme.com/www.mpich.org/ says >> >> *It's not just you! http://www.mpich.org <http://www.mpich.org> looks >> down >> from here.* >> >> Given what I tried to access was >> https://www.mpich.org/static/docs/v3.2/www3/MPI_Win_fence.html, perhaps >> someone can figure out how to mirror the MPICH man pages on Github pages. >> They can even be auto-generated by Travis CI, although I can't say that >> I've done the auto-deployment stuff in Travis before. >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> >> >> >> _______________________________________________ >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/devel >> > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel

1 0

Re: [mpich-devel] mpich.org is downforeveryone[andnotjustme].com
by Halim Amer 08 Mar '17

08 Mar '17

I get a different response than you, Jeff. The site says mpich.org is up. Halim www.mcs.anl.gov/~aamer On 3/8/17 7:37 AM, Jeff Hammond wrote: > http://www.downforeveryoneorjustme.com/www.mpich.org/ says > > *It's not just you! http://www.mpich.org <http://www.mpich.org> looks down > from here.* > > Given what I tried to access was > https://www.mpich.org/static/docs/v3.2/www3/MPI_Win_fence.html, perhaps > someone can figure out how to mirror the MPICH man pages on Github pages. > They can even be auto-generated by Travis CI, although I can't say that > I've done the auto-deployment stuff in Travis before. > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > > > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel >

1 0

mpich.org is downforeveryone[andnotjustme].com
by Jeff Hammond 08 Mar '17

08 Mar '17

http://www.downforeveryoneorjustme.com/www.mpich.org/ says *It's not just you! http://www.mpich.org <http://www.mpich.org> looks down from here.* Given what I tried to access was https://www.mpich.org/static/docs/v3.2/www3/MPI_Win_fence.html, perhaps someone can figure out how to mirror the MPICH man pages on Github pages. They can even be auto-generated by Travis CI, although I can't say that I've done the auto-deployment stuff in Travis before. Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] devel Digest, Vol 44, Issue 1
by Giuseppe Congiu 08 Mar '17

08 Mar '17

Hello Rob, > I'm excited to see someone using the MPIX_Grequest interface. We used > the MPIX_Grequest interface to implement non-blocking collective I/O, > and had some bad interactions between libc's aio and the grequest > callbacks. I don't know if you are running into something similar. > Maybe. Do you have a description of the problem somewhere? > Do you have any desire or plans to submit these changes into upstream > ROMIO? The idea would be to push these changes to upstream ROMIO if this is relevant for the community. > > The BeeGFS ROMIO driver uses the cache API internally provided by > > BeeGFS to move data around, while the custom ROMIO implementation > > provides support for any other file system (in the "common" layer). > > > > Description of the problem: > > I am currently having problems making the BeeGFS driver work > > properly. More specifically I am noticing that when the size of the > > shared file is large (32GB in my configuration) and I am writing > > multiple files one after the other (using a custom coll_perf > > benchmark) the i-th file (for i > 0 with 0 <= i < 4) gets stuck. > > > > BeeGFS provides a deeper_cache_flush_range() function to flush data > > in the cache to the global file. This is non blocking and will submit > > a range (offset, length) for a certain filename to a cache deamon > > that transfers the data. Multiple ranges can be submitted one after > > the other.? > > > > Completion can be checked using deeper_cache_flush_wait() using the > > filename as input. Since the MPI_Grequest model requires the external > > thread to make progress by invoking MPI_Grequest_complete() I have > > opted for the MPIX_Grequest interface which allows me to make > > progress (MPI_Grequest_complete) inside MPI while invoking the > > MPI_Wait() function. > > The original intent of the MPIX_Grequest extensions is that something > else (in this case the AIO library) was making progress in the > background and all we needed was a way to check if things were done or > not. I suspect the callbacks could do some work, but blocking > callbacks haven't seen a lot of testing (in 10+ years the extensions > themselves haven't seen a lot of activity, either) > In principle here I have the same intent. The difference is that I cannot check on progress since BeeGFS does not provide a way for checking the status of a single request. Instead it only offers a blocking wait interface for all the requests submitted for a certain file (identified by the filename). Thus I need to invoke deeper_cache_flush_wait() from inside one of the callbacks. If this is a problem I can revert to MPI_Grequest but I need to spawn a thread that launches deeper_cache_flush_range() for every new request and then waits in deeper_cache_flush_wait() for all submitted requests to complete before calling MPI_Grequest_complete() this should work but is adding an unnecessary thread. > > > > deeper_cache_flush_range() does not require the application to keep > > any handler for the submitted request. The only thing the application > > needs is the name of the file to pass to deeper_cache_flush_wait(). > > > > Follows a code snippet for the corresponding non blocking > > implementation in ROMIO: > > > > /* arguments for callback function */ > > struct callback { > > ? ? ADIO_File fd_; > > ? ? ADIOI_Sync_req_t req_; > > }; > > > > /* > > ?* ADIOI_BEEGFS_Sync_thread_start - start synchronisation of req > > ?*/ > > int ADIOI_BEEGFS_Sync_thread_start(ADIOI_Sync_thread_t t) { > > ? ? ADIOI_Atomic_queue_t q = t->sub_; > > ? ? ADIOI_Sync_req_t r; > > ? ? int retval, count, fflags, error_code, i; > > ? ? ADIO_Offset offset, len; > > ? ? MPI_Count datatype_size; > > ? ? MPI_Datatype datatype; > > ? ? ADIO_Request *req; > > ? ? char myname[] = "ADIOI_BEEGFS_SYNC_THREAD_START"; > > > > ? ? r = ADIOI_Atomic_queue_front(q); > > ? ? ADIOI_Atomic_queue_pop(q); > > > > ? ? ADIOI_Sync_req_get_key(r, ADIOI_SYNC_ALL, &offset, > > ? ?&datatype, &count, &req, &error_code, &fflags); > > > > ? ? MPI_Type_size_x(datatype, &datatype_size); > > ? ? len = (ADIO_Offset)datatype_size * (ADIO_Offset)count; > > > > ? ? retval = deeper_cache_flush_range(t->fd_->filename, > > (off_t)offset, (size_t)len, fflags); > > > > ? ? if (retval == DEEPER_RETVAL_SUCCESS && ADIOI_BEEGFS_greq_class == > > 1) { > > MPIX_Grequest_class_create(ADIOI_BEEGFS_Sync_req_query, > > ?? ? ? ? ? ? ? ? ? ADIOI_BEEGFS_Sync_req_free, > > ?? ? ? ? ? ? ? ? ? MPIU_Greq_cancel_fn, > > ?? ? ? ? ? ADIOI_BEEGFS_Sync_req_poll, > > ?? ? ? ? ? ADIOI_BEEGFS_Sync_req_wait, > > ?? &ADIOI_BEEGFS_greq_class); > > ? ? } else { > > /* --BEGIN ERROR HANDLING-- */ > > return MPIO_Err_create_code(MPI_SUCCESS, > > ?? ? ? ? ? ? ? ? ? ?MPIR_ERR_RECOVERABLE, > > ?? ? ? ? ? ? ? ? ? ?"ADIOI_BEEGFS_Cache_sync_req", > > ?? ? ? ? ? ?__LINE__, MPI_ERR_IO, "**io %s", > > ?? ? ? ? ? ?strerror(errno)); > > ? ? ? ? /* --END ERROR HANDLING-- */ > > ? ? } > > > > ? ? /* init args for the callback functions */ > > ? ? struct callback *args = (struct callback > > *)ADIOI_Malloc(sizeof(struct callback)); > > ? ? args->fd_ = t->fd_; > > ? ? args->req_ = r; > > > > ? ? MPIX_Grequest_class_allocate(ADIOI_BEEGFS_greq_class, args, req); > > > > ? ? return MPI_SUCCESS; > > } > > > > /* > > ?* ADIOI_BEEGFS_Sync_req_poll - > > ?*/ > > int ADIOI_BEEGFS_Sync_req_poll(void *extra_state, MPI_Status *status) > > { > > ? ? struct callback *cb = (struct callback *)extra_state; > > ? ? ADIOI_Sync_req_t r = (ADIOI_Sync_req_t)cb->req_; > > ? ? ADIO_File fd = (ADIO_File)cb->fd_; > > ? ? char *filename = fd->filename; > > ? ? int count, cache_flush_flags, error_code; > > ? ? MPI_Datatype datatype; > > ? ? ADIO_Offset offset; > > ? ? MPI_Aint lb, extent; > > ? ? ADIO_Offset len; > > ? ? ADIO_Request *req; > > > > ? ? ADIOI_Sync_req_get_key(r, ADIOI_SYNC_ALL, &offset, > > ? ?&datatype, &count, &req, &error_code, &cache_flush_flags); > > > > ? ? int retval = deeper_cache_flush_wait(filename, > > cache_flush_flags); > > > > ? ? MPI_Type_get_extent(datatype, &lb, &extent); > > ? ? len = (ADIO_Offset)extent * (ADIO_Offset)count; > > > > ? ? if (fd->hints->e10_cache_coherent == ADIOI_HINT_ENABLE) > > ADIOI_UNLOCK(fd, offset, SEEK_SET, len); > > > > ? ? /* mark generilized request as completed */ > > ? ? MPI_Grequest_complete(*req); > > > > ? ? if (retval != DEEPER_RETVAL_SUCCESS) > > ? ? ? ? goto fn_exit_error; > > > > ? ? MPI_Status_set_cancelled(status, 0); > > ? ? MPI_Status_set_elements(status, datatype, count); > > ? ? status->MPI_SOURCE = MPI_UNDEFINED; > > ? ? status->MPI_TAG = MPI_UNDEFINED; > > > > ? ? ADIOI_Free(cb); > > > > ? ? return MPI_SUCCESS; > > > > fn_exit_error: > > ? ? ADIOI_Free(cb); > > > > ? ? return MPIO_Err_create_code(MPI_SUCCESS, > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? MPIR_ERR_RECOVERABLE, > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? "ADIOI_BEEGFS_Sync_req_poll", > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? __LINE__, MPI_ERR_IO, "**io %s", > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? strerror(errno)); > > } > > > > /* > > ?* ADIOI_BEEGFS_Sync_req_wait - > > ?*/ > > int ADIOI_BEEGFS_Sync_req_wait(int count, void **array_of_states, > > double timeout, MPI_Status *status) { > > ? ? return ADIOI_BEEGFS_Sync_req_poll(*array_of_states, status); > > } > > > > /* > > ?* ADIOI_BEEGFS_Sync_req_query - > > ?*/ > > int ADIOI_BEEGFS_Sync_req_query(void *extra_state, MPI_Status > > *status) { > > ? ? return MPI_SUCCESS; > > } > > > > /* > > ?* ADIOI_BEEGFS_Sync_req_free - > > ?*/ > > int ADIOI_BEEGFS_Sync_req_free(void *extra_state) { > > ? ? return MPI_SUCCESS; > > } > > > > /* > > ?* ADIOI_BEEGFS_Sync_req_cancel - > > ?*/ > > int ADIOI_BEEGFS_Sync_req_cancel(void *extra_state, int complete) { > > ? ? return MPI_SUCCESS; > > } ? > > > > I have had a look at the implementation for MPI_File_iwrite() in > > ad_iwrite.c but this uses POSIX AIO, thus I am not sure I am doing > > things properly here. > > the poll_fn() should be non-blocking. think MPI_Test more than unix > poll/select/epoll. it's going to get called in a tight loop. > So calling a blocking function inside the poll callback is not advised. I am currently referring to "Extending the MPI-2 Generalised Request Interface" paper and some iwrite example in ROMIO for my implementation. Is there a more detailed document for MPIX_Grequest? Thank you, -- Giuseppe Congiu *·* Research Engineer II Seagate Technology, LLC office: +44 (0)23 9249 6082 <+44%2023%209249%206082> *·* mobile: www.seagate.com

1 0